AD-A108  499 
UNCLASSIFIED 


CANYON  RESEARCH  GROUP  INC  WESTLAKE  VILLAGE  CA  p/G  5/o 

APPLICATION  OF  A  MULTIFACTOR  APPROACH  TO  TRANSFER  OF  TRAINING  R _ ETC(U) 

^  2}  aS  "  SIM0N'  S  N  R0SC0E  N6I339-78-C-0060 

CWS-01-82  NAVTRAEQUIPC-78-C-0060-6 


NL 


r 


AD  A1v)B499 


<g)  LEVELS 


Technical  Report  NAVT1AEQUIPCEN  78-C-0060-6 


APPLICATION  OF  A  MDLTIFACTOR  APPROACH 
TO  TRANSFER  OF  TRAINING  RESEARCH 


c  cv. 


Charles  W.  Siaon 

Canyon  Research  Group,  Inc. 

741  Lakefleld  Road,  Suite  B 
Westlake  Villas*.  California  91361 


Stanley  N.  Roscoe 

ULIANA  Aviation  Sciences,  Ltd. 

2090  Vyosdng  Avenue 

Las  Cruces,  New  Mexico  88001 


s 


DTIC 

ELECTE| 
DEC  1  0  1981 


TECHNICAL  REPORT  MAT  1980  -  July  1981 
July  1981 


DoD  Distribution  Statement 

Approved  for  public  release; 
distribution  uni ini ted. 


EQU 


ENT 


NTE 


|V  3 


*p  P 


81  12  10  01? 


■AvniiqpxFcm  78-C-0060-6 


cggnagB  lions  d»  mta  staibipit 

Reproduction  of  this  publication  in 
whold  or  In  part  Is  permitted  for  any 
purpose  of  Idas  United  States 


SECURITY  CLASSIFICATION  or  Thu  page  (Whm  Oata^tMmod) 


*  REPORT  DOCUMENTATION  PAGE 

READ  INSTRUCTIONS 

BEFORE  COMPLETING  FORM 

1.  AlloiVNUykER  2.  OOVT  ACCESSION  NO. 

NAVTRAEQUIP)feEN  78-C-0060-6  -  -j  j  C  %  y 

1.  NECIPIENT'S  CATALOG  NUMBER 

ft 

4.  TITLE  (mot  luMllaJ 

APPLICATION  OF  A  MULTIFACTOR  APPROACH 

TO  TRANSFER  OF  TRAINING  RESEARCH 

3.  TYPE  OF  REPORT  A  PERIOD  COVERED 

Technical  Report 

1  May  1980  -  31  July  1981 

S.  PERFORMING  ORO.  REPORT  NUMBER 

CVS  01-82 

7.  AUTHOR*) 

Charles  W.  Simon  and  Stanley  N.  Roscoe 

A.  CON CRACV  OR  GRANT  HUMBERT*; 

N  61339-78-C-0060 

S.  FCNFORMINO  ONOANIZATION  NAME  AND  AOONESS 

Canyon  Research  Group,  Inc. 

741  Lakefield  Road,  Suite  B 

Westlake  Village,  CA  91361 

10.  PROGRAM  ELEMENT.  PROJECT.  TASK 
AREA  A  WORK  UNIT  NUMBERS 

NAVTRAEQUIPCEN 

Project  No.  4781-6P1A 

II.  CONTHOLLING  OFFICE  NAME  AND  AOONESS 

Naval  Training  Equipment  Center 

Orlando ,  Florida  32813 

12.  REPORT  OATE 

July  1981 

IS.  NUMBER  OF  PAGES 

_ 82 _ 

14  MONITOHING  AGENCY  NAME  A  AOORES3TM  dfAiail  from  Controlling  Ottteo) 

IS.  SECURITY  CLASS,  (ol  thlo  report) 

Unclassified 

is «.  declassification/ OOWNGR AOING 
scheoule 

U.  DISTRIBUTION  ST ATCMCNT  (of  thla  Report) 

Approved  for  public  release;  distribution  unlimited* 

17.  OISTNINUTION  STATEMENT  (at  Hit  abound  ontotad  In  Block  20,  II  dll  lor  on  I  Irom  Report) 

IS.  SUPPLEMENTARY  NOTES 

If.  KEY  WORDS  fConifnuo  on  rororao  aldo  If  nocoaaaay  and  Identify  by  block  mmbor) 

Training  and  transfer  data  relationships  Transfer  of  training 

Economical  multifactor,  multicriterion  study  Experimental  designs 

Visual  Technology  Research  Simulator  (VTRS)  Research  methods 

are  expensive  to  perform 
the  extended  training 
sting.  More  economical 
the  mission  of  the  Naval 
earch  Simulator.— 7, 

'  'Jr~ 

(over) 

Multifactor  transfer  of  training  experiments 
because  of  the  large  number  of  subjects  required, 
they  receive,  and  their  subsequent  operational  te 
data  collection  techniques  are  needed  to  fulfill 
Training  Equipment  Center's  Visual  Technology  Res 

00  1473  COITION  OF  <  NOV  SS  It  OBSOLETE 


Unclassified _ 

tCCUNITV  CLASSIFICATION  OF  THIS  FAOC  (Whit  Om  Rotated) 


seeuWITY  Cl.*»SiriCATIOH  OF  THIS  PAOKfWh^  Dim  Kittmr*d)  _ _ 

20.  Continued 

I 

\ 

[  An  experimental  effort  was  undertaken:  (a)  to  establish  relationships 

among  training,  test,  and  transfer  scores  in  the  context  of  the  manual  con¬ 
trol  of  a  maneuvering  vehicle',  ChT  to  determine  the  relative  complexities  of 
response  surfaces  for  training,  test,  and  transfer;  (cV  to  demonstrate  a  new 
transfer  research  paradigm  that  makes  economically  feasible  the  simultaneous 
investigation  of  the  effects  of  a  large  number  of  equipment  design  variables 
on  the  transfer  to  multiple  test  configurations;  and  (df  to  extract  from  the 
available  data  indications  that  will  enable  the  transfer  effectiveness  of 
simulator  characteristics  to  be  estimated  with  minimum  costs., 

A  horizontal  tracking  task  was  used  in  the  study.  Six  factors  were 
varied  to  form  49  simulator  training  configurations.  These  factors  included 
five  dynamic  simulator  design  variables:  vehicle  control  order,  display  lag, 
tracking  mode  (pursuit  vs.  compensatory) ,  prediction  time,  and  control  gain. 
The  sixth  variable  was  the  number  of  training  trials  given  before  transition 
to  one  of  three  transfer  vehicle  configurations,  designated  Hard,  Central, 
and  Easy. 

Eighty  flight-naive  adult  males  participated  in  the  experiment.  Each  of 
48  was  trained  and  tested  on  a  different  combination  of  training  and  transfer 
simulator  configurations.  Eight  more  were  trained  and  tested  on  the  same 
49th  combination.  Data  collected  at  these  49  points  provided  estimates  of 
all  main  effects  and  two-factor  interactions  for  both  the  training  and  trans¬ 
fer  configurations.  Additional  data  were  collected  from  three  control  groups 
who  received  no  prior  training. 

>  In  this  initial  experiment  the  following  relationships  between  training 
and  transfer  performance  were  indicated:  (*)  the  transfer  surface  appeared 
less  complex  that  “'the  training  surface;  (b)  the  relationship  between  training 
and  transfer  scores  was  positive  but  too  weak  for  predictive  purposes,  (c) 
some  factors  had  large  effects  in  training  and  small  effects  in  transfer,  and 
vice  versa',  and  Cd^  transfer  was  facilitated  when  the  values  of  certain 
variables  result  in  training  conditions  that  were  more  difficult  than  the 
subsequent  transfer  criterion  conditions. ^ Future  investigations  will  help 
determine  their  generality. 

The  study  demonstrated  the  efficiency  and  economy  in  collecting  multi¬ 
factor,  multicriterion  transfer  of  training  data.  This  type  of  experiment 
is  particularly  useful  in  the  early  stages  of  a  simulator  design  program 
when  many  alternatives  should  be  considered  and  the  individual  contributions 
of  component  design  parameters  should  be  evaluated  separately  from  overall 
simulator  effectiveness. 
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FOREWORD 


One  of  the  great  drawbacks  associated  with  training  device  and 
training  technology  research  is  the  difficulty  of  conducting  transfer 
of  training  experiments.  The  reasons  are  well-known:  for  instance, 
research  that  involves  training  people  to  use  operational  equipment 
is  costly,  complicated  ar\d  logistically  difficult;  the  use  of  a  large 
number  of  subject,  participants  is  usually  necessary;  and  very  few 
experimental  factors  can  be  examined  at  one  time.  The  result:  a  great 
deal  of  money  is  spent  to  do  relatively  few  experiments,  and  our 
progress  in  understanding  issues  of  trainer  design  and  usage  is  slow. 

This  report  is  concerned  with  two  means  by  which  progress  may  be 
accelerated:  (1)  using  more  efficient  experimental  designs  in  order  to 
examine  more  factors  simultaneously;  and  (2)  understandng  the  relation¬ 
ships  between  performance  and  transfer  data  for  different  classes  of 
variables,  in  order  to  predict  transfer  on  the  basis  of  performance 
data.  The  first  author,  Dr.  Simon,  has  worked  for  more  than  a  decade 
to  study  and  publicize  methods  for  conducting  more  informative  and 
efficient  performance  experiments.  The  second  author,  Dr.  Roscoe,  has 
devoted  most  of  his  professional  life  to  aviation  training  research. 
They  have  collaborated  to  produce  the  first  reported  attempt  to  use  an 
economical  multifactor  design  to  explore  issues  of  training  and 
transfer. 

Psychologists  involved  in  applications-oriented  simulation  and 
training  research  will  find  this  report  thought  provoking,  both  in 
terms  of  metholodogy  and  results.  Considering  the  number  of  variables 
examined,  it  was  the  most  complex  transfer  of  training  experiment  ever 
done,  and  serves  to  show  that  it  is  no  longer  necessary  to  think  solely 
in  terms  of  small-factor  studies  —  there  are  alternatives.  The 
results  provide  some  information  on  a  variety  of.  performance-transfer 
relationships  (e.g.,  relative  complexity  of  surfaces,  factor  fidelity, 
relative  difficulty). 

Because  this  was  a  preliminary  effort,  the  experimental  apparatus 
was  of  only  moderate  complexity,  and  the  factors  manipulated  are  not 
directly  relevant  to  issues  of  simulator  design.  However,  much  infor¬ 
mation  obtained  in  the  course  of  this  work  has  been  used  to  prepare  for 
an  experiment  now  underway  on  the  Naval  Training  Equipment  Center's 
Visual  Technology  Research  Simulator  (VTRS).  That  work,  investigating 
seven  design,  subject  and  difficulty  factors,  will  be  published  in  1981 
as  NAVTRAEQUIPCEN  78-C-0060- 1 2 . 


STANLEY  C.  COLLYER 
Scientific  Officer 
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SUMMARY 


In  designing  the  experiaent,  therefore,  the  aim 
of  the  experimenter  should  usually  be,  not  to 
provide  for  the  highest  possible  degree  of  pre¬ 
cision  in  the  estimate,  but  rather  to  secure, 
with  the  minimum  expenditure  of  his  resources, 
whatever  degree  of  precision  and  freedom  from 
bias  is  sufficient  for  his  purposes. 

E.  F.  Lindquist,  Design  and 
Analysis  of  Experiments  in 
Psychology  and  Education.  1953 


The  Visual  Technology  Research  Simulator  of  the  Research  Depart¬ 
ment,  Naval  Training  Equipment  Center,  Orlando,  Florida,  was  created  to 
support  the  research  objectives  of  two  groups:  the  engineers,  con¬ 
cerned  with  improving  visual  system  technology,  and  the  psychologists, 
concerned  with  evaluating  that  technology  from  the  standpoint  of  train¬ 
ing  effectiveness.  On  the  one  hand,  the  VTRS  serves  as  a  test  bed  for 
developing  and  evaluating  new  engineering  concepts  aimed  at  providing 
increased  visual  system  performance  at  lower  costs.  At  the  same  time, 
it  is  a  research  tool  for  examining  how  pilot  performance  and  transfer 
of  training  are  Influenced  by  a  wide  range  of  factors  of  critical 
interest  to  simulation  engineers  and  training  specialists  (Collyer  and 
Chambers,  1978). 

Traditional  experiments  for  measuring  performance  and  transfer  of 
training  are  costly  to  conduct  and  limited  in  the  amount  of  information 
they  generate  concerning  the  large  number  of  factors  of  interest  in  the 
design  of  training  simulators.  New  approaches  that  provide  more  infor¬ 
mation  at  less  cost  have  been  tried  in  recent  years,  but  are  still 
costly  by  practical  standards.  For  an  effective  research  effort,  more 
economical  methods  particularly  suited  to  transfer  of  training  research 
must  be  developed. 

The  objectives  of  this  study  were  (a)  to  establish  relationships 
among  training,  test,  and  transfer  scores  in  the  context  of  a  complex 
manual  tracking  task,  (b)  to  determine  the  relative  complexities  of 
response  surfaces  for  training,  test,  and  transfer,  (c)  to  demonstrate 
a  new  research  paradigm  for  the  simultaneous  investigation  of  the 
effects  of  a  large  number  of  simulator  design  variables  on  transfer  to 
different  vehiole  configurations,  and  (d)  to  extract  from  the  available 
data  indications  that  will  enable  the  transfer  effectiveness  of  simu¬ 
lator  characteristics  to  be  estimated  with  minimum  costs. 


The  data  for  the  study  were  collected  at  New  Mexico  State  Univer¬ 
sity  (NMSU)  by  personnel  of  ILLIANA  Aviation  Sciences.  The  experimen¬ 
tal  task  involved  the  lateral  tracking  of  a  target  following  a  con¬ 
stantly  changing  course  generated  by  a  microcomputer  and  presented  by 
appropriate  display  symbols  indicating  target  position,  aircraft 
position  and  attitude,  and  a  predicted  aircraft  position.  Dynamic 
simulator  design  variables  that  were  systematically  manipulated 
included:  (a)  vehicle  control  order,  (b)  display  lag,  (c)  tracking 
mode  (pursuit  vs.  compensatory),  (d)  prediction  time,  and  (e)  control 
gain.  A  sixth  variable  was  the  number  of  training  trials  before 
transition  to  one  of  three  transfer  vehicle  configurations,  designated 
Hard,  Central,  and  Easy. 

In  all,  80  flight-naive  young  male  adults  participated  in  the 
experiment.  Of  these,  48  were  assigned  to  three  experimental  groups  of 
16  each  and  were  trained  under  48  uniquely  different  simulator  configu¬ 
rations.  After  training,  the  three  groups,  respectively,  transferred 
to  Hard,  Central,  and  Easy  transfer  vehicle  configurations.  The  48 
unique  combinations  of  training  and  transfer  conditions  were  selected 
so  that  the  effects  of  all  factors,  including  transfer  vehicle  con¬ 
figuration,  and  all  two-factor  interactions  could  be  isolated.  A  49th 
data  point  was  added  at  the  center  of  the  experimental  space  on  which 
eight  more  participants  were  trained  and  subsequently  transferred  to 
the  Central  transfer  vehicle  configuration.  These  extra  data  allowed 
tests  of  the  adequacy  of  the  second-order  model  and  for  curvilinearity . 
Three  control  groups  of  eight  participants  each  were  tested  on  the 
transfer  vehicle  configurations  without  prior  training. 

The  data  were  analyzed  by  personnel  of  the  Canyon  Research  Group 
in  California  and  Florida  and  were  interpreted  by  the  authors.  The 
study  demonstrated  a  unique  and  economical  means  of  performing  holistic 
transfer  of  training  research  and  provided  useful  information  regarding 
relationships  among  training  and  transfer  performances  under  an  unpre¬ 
cedented  number  of  conditions.  The  information  obtained  is  being 
applied  to  the  design  of  transfer  of  training  experiments  to  be  con¬ 
ducted  in  the  VTRS  laboratory. 

This  experiment  was  the  first  to:  (a)  study  the  effects  of  as 
many  as  six  equipment  and  training  factors  in  a  single  transfer  of 
training  experiment,  (b)  examine  a  broad  spectrum  of  training  vehicle 
configurations— 49 — in  the  same  experiment,  (c)  train  only  a  single 
participant  on  each  of  48  training  vehicle  configurations,  (d)  employ 
multiple  transfer  configurations — 3— in  the  same  experiment,  and  (e) 
provide  data  suitable  for  deriving  multiple  regression  equations  for 
estimating  the  effectiveness  of  configurations  not  directly  studied. 

This  oultifactor,  multivariate  economical  data  collection  plan 
demonstrates  the  effectiveness  of  the  approach,  particularly  applicable 
in  the  preliminary  design  stage  of  a  simulator  engineering  program  when 
the  relative  merits  of  a  large  number  of  simulator  configurations  must 


be  compared  to  establish  basic  characteristics.  Furthermore,  the 
multif actor  design  reveals  each  component's  contribution  to  total  simu¬ 
lator  transfer  effectiveness,  permitting  sources  of  negative  transfer 
to  be  identified  even  though  overall  'transfer  is  positive. 

The  following  results  would  appear  to  have  transfer  of  training 
applications  provided  subsequent  investigations  demonstrate  their 
generality: 


1 .  The  transfer  surface  was  less  complex  than  the  training 
surface. 

2.  The  correlation  between  training  and  transfer  performance 
for  different  design  configurations  was  positive  but  too 
weak  for  predictive  purposes. 

3.  Some  factors  had  large  effects  in  training  but  not  in 
transfer,  whereas  some  with  small  effects  in  training  had 
relatively  large  transfer  values. 

4.  Transfer  was  facilitated  when  the  values  of  certain  variables 
resulted  in  training  conditions  that  were  more  difficult  than 
the  subsequent  transfer  criterion  conditions. 

The  cost  of  conducting  this  six-factor,  three- test-configuration 
study  in  terms  of  subject  numbers  and  hours  of  data  collection  was  less 
than  one-eighth  of  that  for  the  same  information  obtained  in  a  series 
of  smaller  experiments  using  conventional  methods,  and  the  data  were 
more  generalizable,  more  precise,  and  less  biased. 
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PREFACE 


A  number  of  persona  made  major  contributions  to  this  experiment. 

Louis  Corl  of  ILLIANA  Aviation  Sciences  was  reponsible  for  the 
hardware  and  software  to  generate,  display,  and  control  the  simulated 
horizontal  steering  task  and  for  the  automatic  data  acquisition  and 
reduction.  Jan  Christopher  Hull,  Paul  M.  Simon,  and  Donald  G. 
Fahrenkrog,  also  of  ILLIANA  Aviation  Sciences,  trained  and  tested  the 
participants. 

The  80  participants  were  volunteers  from  the  local  community  of 
Las  Cruces,  New  Mexico,  and  from  the  Department  of  Psychology,  New 
Mexico  State  University. 

Daniel  P.  Westra  of  the  Canyon  Research  Group  helped  analyze  and 
interpret  the  experimental  data,  and  Brian  Nelson  of  the  Canyon 
Research  Group  helped  prepare  special  computer  software  for  some 
analyses. 

Linda  Carlson-Wenger  of  ILLIANA  Aviation  Sciences  processed  the 
manuscript. 
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SECTION  I 


INTRODUCTION 

Transfer  research  to  evaluate  pilot  training  simulators  has  been 
underway  for  more  than  30  years.  Williams  and  Flexman  (19**9)  reported 
the  first  such  experiment  in  19*19.  Still  there  are  few  simulator  and 
training-program  design  principles  that  experts  earn  agree  on,  and  even 
those  widely  accepted  cannot  be  stated  with  sufficient  precision  to 
•support  choices  of  design  parameters  with  confidence  (Roscoe,  1980, 
Chapters  15-22;  NATO-AGARD,  1980). 

There  are  many  reasons  for  our  limited- understanding  of  the 
dependence  of  training  effectiveness  on  simulator  and  training-program 
design.  At  best,  transfer  experiments  involving  complex  simulators  and 
real  airplanes  are  loglstically  difficult  and  costly.  Furthermore, 
classical  experimental  designs  (Campbell  and  Stanley,  1963),  while 
suitable  for  evaluating  particular  devices  or  systems,  are  less  suit¬ 
able  for  research  Intended  to  Identify  which  of  a  great  many  variables 
in  training  positively  affect  performance  of  the  same  task  in  the 
operational  situation. 

In  the  real  world  of  pilot  training,  the  number  of  potentially 
important  simulator  and  training  variables  is  disturbingly  large.  They 
fall  into  various  classes  and  often  interact  to  a  degree  that  defies 
comprehensive  analysis.  To  study  this  large  number  of  variables  using 
con.entional  approaches  has  proved  unacceptably  expensive,  if  not  im¬ 
possible.  To  consider  only  a  few  at  a  time  produces  results  that  are 
inevitably  inaccurate  when  applied  to  operational  situations  and, 
because  of  the  situation-specific  nature  of  human  behavior,  cannot  be 
generalized  with  confidence  far  from  the  values  fixed  in  the  experiment. 

The  problem  of  conducting  transfer  of  training  studies  would  be 
greatly  simplified  if  one  could  realistically  assume  that  transfer  of 
simulator  training  to  actual  flight  could  be  predicted  from  relative 
performances  in  the  simulator.  However,  one  can  easily  conceive  of 
situations  in  which  features  that  make  a  simulator  easier  to  fly  will 
not  necessarily  promote  higher  transfer  to  the  operational  situation. 
Unfortunately  there  have  been  few  systematic  investigations  of  the 
relationship  between  performance  during  training  and  transfer  of 
perceptual -motor  skills.  Until  relationships  have  been  established 
empirically  and  can  be  expressed  quantitatively,  transfer  experiments 
will  be  required.  Finding  practical  ways  of  collecting  good  information 
inexpensively  regarding  the  effect  of  training-simulator  design 
variables  is  a  prerequisite  to  the  collection  of  such  data. 
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SECTION  II 
APPROACH 

Recognizing  that  a  problem  exists  ia  the  firat  atep  in  any  scien- 
tiflc  inquiry.  Becauae  it  ia  evident  that  the  problem  of  obtaining  good 
oultifactor  training-simulator  design  information  ia  both  economic  and 
logistic,  a  "holistic"  approach  to  experimentation  haa  been  adopted.  The 
term  holistic  refers  to  "a  philosophic  point  of  view  in  the  conduct  of 
behavioral  experiments  that  emphasizes  the  importance  of  accounting  for 
aa  many  potentially  critical  variables  aa  possible,  whether  equip¬ 
ment,  environment,  subject,  or  temporal,  controlled  or  uncontrolled" 
(Simon,  1979,  p.  77).  Human  behavior  cannot  be  isolated  to  the  same 
degree  that  chemical  compounds  can  be.  It  is  situation  specific  and 
must  be  defined  in  terms  of  the  total  situation. 

The  key  word,  in  the  above  definition  is  "critical."  Whatever  their 
number,  if  critical  variables  are  held  constant  in  an  experiment,  unless 
the  fixed  values  are  close  to  those  found  operationally,  findings  can  be 
grossly  inaccurate  when  applied  to  the  operational  situation.  Properly 
implemented,  the  holistic  approach  will  yield  data  that  are  more  pre¬ 
cise,  less  biased,  and  more  generallzable  from  laboratory  to  field  for 
far  less  cost  than  the  traditional  elemental  approach  (Simon,  1979, 
Chapter  II).  In  an  ongoing  research  program,  the  savings  in  time  and 
resources  can  be  considerable. 

The  use  of  economical  multifactor  designs  and  sequential  strate¬ 
gies  (Simon,  1973;  1977)  can  significantly  reduce  the  cost  of  doing 
holistic  experiments.  Recently,  this  approach  was  successfully 
implemented  in  a  study  of  performance  during  carrier  landing  as  a 
function  of  ten  equipment/environment  factors  (Westra,  et  al.,  1981). 
However,  when  many  factors  must  be  investigated,  these  economical 
designs  may  still  not  be  sufficiently  economical  if  extended  training 
is  involved.  That  is  to  say,  even  with  a  significant  reduction  in  the 
number  of  data  points  sampling  the  coordinate  space,  compared  with  the 
number  called  for  by  traditional  factorial  designs,  the  size  of  the 
effort  may  still  be  impractical  to  consider. 

Such  designs  may  be  impractical  for  transfer  of  training  studies 
because,  for  each  data  point,  one  or  more  subjects  must  be  trained  for 
an  extended  period  and  subsequently  tested  during  a  transfer  period. 

In  some  cases,  still  more  data  must  be  collected  for  control  groups. 
Furthermore,  the  nature  of  a  transfer  study  requires  that  a  different 
subject  (or  subjects)  be  tested  at  each  data  point,  a  condition  that  is 
not  always  necessary  for  performance  studies  in  which  each  subject  may 
be  tested  on  several  conditions. 


For  these  reasons,  unmodified  economical  multifactor  designs  may 
still  not  be  sufficiently  economical  for  transfer  of  training  research, 
but  what  are  the  alternatives?  An  investigator  may  again  be  tempted  to 
resort  to  a  series  of  small  studies  involving  a  few  factors  at  a  time. 
But  this  will  not  solve  the  problem  since,  for  any  given  number  of  fac¬ 
tors  and  for  an  eqyal  degree  of  precision,  it  is  less  costly  to  study 
them  all  in  a  single  experiment  than  in  any  series  of  smaller  experi¬ 
ments  that  requires  expensive  replication  for  precision. 

This  does  not  deny  the  appropriate  use  of  small  experiments  in  a 
holistic  approach,  but  they  must  be  done  in  a  manner  and  context  that 
permit  the  results  of  each  to  be  included  in  the  ever  expanding  overall 
data  base.  Merely  reducing  the  number  of  factors  to  reduce  cost  also 
is  not  a  reasonable  solution  if  all  are  potentially  important  and  their 
relative  importance  is  unknown. 

Because  so  little  is  known  about  the  characteristics  of  a  multi¬ 
factor  transfer  of  training  performance  space,  or  the  relationships 
among  training,  transfer,  and  intervening  variables,  the  present  study 
was  designed  to  test  a  particularly  economical  transfer  of  training  data 
collection  plan  and  to  use  the  resulting  data  to  investigate  the  rela¬ 
tionships  between  training  and  transfer  performance. 
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SECTION  III 

THE  DATA  COLLECTION  PLAN 

It  was  decided  at  the  start  that  an  artificial  data  base  would  not 
be  appropriate  to  use.  Even  with  a  limited  task,  it  was  thought  that 
the  characteristic  relationships  between  training  and  transfer  would  be 
more  faithfully  represented  by  an  actual  data  base  than  by  a  struc¬ 
tured,  computer-generated  base  that  could  only  reflect  the  character¬ 
istics  introduced  by  the  investigator.  While  the  use  of  a  relatively 
simple  task  was  accepted  as  a  limitation  of  this  exploratory  study,  it 
corresponds  to  some  degree  with  those  of  primary  interest  in  the  real 
world  and  the  simulated  system  should  involve  variables  important  in 
the  design  of  operational  simulators. 

Simulator 


Lateral  control  of  a  vehicle  having  simplified  dynamic  responses 
similar  to  an  airplane  was  programed  on  the  MicroGraphic  Simulator  of 
the  Behavioral  Engineering  Laboratory,  New  Mexico  State  University. 
This  versatile  research  facility  consists,  in  part,  of  an  ADAC  System 
1000  microcomputer,  a  plasma-panel  matrix  display,  and  a  three-axis 
manual  flight  controller,  only  the  lateral  dimension  of  which  was  used 
in  this  simulation.  The  system  also  performs  automatic  performance 
scoring  and  immediate  data  reduction,  partial  analysis,  and  visual 
display,  with  subsequent  printout  on  a  daisy-wheel  printer. 

Task 


The  tracking  task  involved  the  relative  lateral  positions  of  three 
symbols  (Figure  1)  on  a  plasma-panel  display  located  approximately  one 
meter  in  front  of  the  seated  participant.  The  three  symbols  appeared, 
respectively,  as  an  airplane-like  figure  viewed  from  the  rear,  a  target 
box,  and  a  small  predictor  circle  (approximated  by  an  octagon).  When¬ 
ever  the  prediction-time  variable  was  set  at  zero,  the  predictor  circle 
was  superposed  concentrically  with  the  vehicle  symbol  and  appeared  as 
if  it  represented  the  body  (or  fuselage)  of  an  airplane. 

Using  a  control  stick  located  on  the  right  arm  of  the  seat,  each 
participant,  regardless  of  the  system  configuration,  was  instructed  to 
manipulate  the  control  to  keep  the  vehicle  and  the  target  as  close  to 
each  other  as  possible  (thus  the  object  was  to  keep  the  vehicle 
directly  superposed  on  the  target  throughout  a  trial).  The  simulated 
vehicle  was  maneuvered  by  right  and  left  deflections  of  the  control 
stick.  If  the  stick  were  deflected  to  the  left,  the  vehicle  responded 
in  kind,  either  banking  and  moving  left  or  Just  banking  left,  depending 
on  whether  the  display  mode  were  pursuit  or  compensatory. 

The  target  was  driven  by  a  forcing  function  (or  course)  generated 
by  the  summation  of  four  sine  waves,  a  fundamental  sinusoid  having  a 
period  of  Ml  sec  and  its  2nd,  5th,  and  13th  harmonics.  Trials  were 
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each  51  seconds  in  length,  of  which  the  first  ten  seconds  were  not 
scored.  For  any  combination  of  system-configuration  variables,  the 
true  tracking  error  between  the  vehicle  and  the  target  was  displayed 
unless  one  or  the  other  were  in  saturation  at  the  left  or  right  edge  of 
the  screen.  Because  scoring  was  based  on  the  displayed  error,  the 
operator  could  see  exactly  what  was  being  scored. 

Experimental  Factors 

The  factors  manipulated  to  form  the  different  simulator  configu¬ 
rations  were:  Control  Order  (CO),  Display  Lag  (DL),  Tracking  Mode 
(TM),  Prediction  Time  (PT),  and  Control  Cain  (CG).  Number  of  Training 
Trials  (TT)  was  a  sixth  experimental  factor.  These  factors  and  their 
levels  under  training  and  transfer  conditions  are  listed  in  Tables  1 
and  2,  respectively.  It  was  intended  that  this  set  of  factors  would  be 
rich  in  a  variety  of  transfer  characteristics,  including  high  and  low 
performances  in  training  and  both  high  and  low  positive  and  negative 
effects  during  transfer.  This  objective  was  not  fully  realized.  The 
understanding  of  the  relationships  required  to  make  3uch  judgments  in 
advance  does  not  yet  exist. 

Control  Order  (CO).  The  dynamic  response  of  a  vehicle  to  control 
inputs  can  range  from  zero-order  to  third-order  or  higher.  In  zero- 
order  the  position  of  the  vehicle  (in  this  case  its  lateral  position) 
corresponds  directly  to  the  deflection  of  the  operator's  control  stick. 
In  first-order  control,  stick  deflection  directly  determines  velocity, 
or  rate,  of  lateral  movement;  in  second-order,  lateral  acceleration; 
and  so  it  goes.  Generally,  the  higher  the  order  the  more  difficult  the 
control,  but  relative  transfer  will  also  depend  on  the  closeness  of  the 
match  between  training  and  transfer  control  dynamics.  For  this  study, 
control  orders  ranged  from  100%  acceleration  (designated  the  Hard 
level)  to  a  combination  of  75%  velocity  and  25%  acceleration  (the  Easy 
level).  The  midrange  (Central)  level  dynamically  combined  38%  velocity 
and  62%  acceleration. 

Display  Lag  (DL).  Time  lags  between  vehicle  response  and  display 
indications  can  have  large  effects  on  both  performance  and  transfer. 
Their  negative  effect  on  simulator  performance  is  similar  to  that  of 
increasing  control  order,  but  display  lags  can  be  beneficial  to 
transfer  if  learning  to  anticipate  vehicle  responses  is  a  major 
component  of  the  task,  as  in  formation  flying.  Typical  lags  can  be 
exponential  in  form,  as  occur  when  noisy  signals  are  smoothed  by 
filtering,  or  simple  transport  delays.  Because  the  latter  are  common 
in  the  updating  of  digitally  generated  visual  displays  in  flight 
simulators,  variable  transport-  lag  was  selected  as  the  second  factor. 
The  three  levels  (-,  0,  and  +)  were  0.30,  0.15,  and  0.00  second, 
respectively. 
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TABLE  1.  TRAINING  PHASE:  EXPERIMENTAL  SYSTEM-CONFIGURATION  AND 
CURRICULUM  FACTORS  AND  THEIR  HARD  (-),  CENTRAL  (0), 

AND  EASY  (+)  VALUES 


LEVELS* 


FACTOR 

- 

0 

+ 

Control  Order  (CO)  in 

percent  acceleration 

100 

62 

25 

Display  Lag  (DL)  in  seconds 

.30 

.15 

.00 

Tracking  Mode  (TM)  in 

percent  pursuit 

0 

50 

100 

Prediction  Time  (?T) 

in  seconds 

.0 

.3 

.6 

Control  Gain  (CG)  in 

unitless  ratios 

.12 

.18 

.24 

Number  of  Training  Trials  (TT) 

10 

20 

30 

•Levels  designated  -  and  ♦  are  actually  coded  -1  and  +1,  respectively, 
corresponding  to  what  was  assumed  a  priori  to  be  the  Hard  and  Easy 
levels.  The  0  is  the  Central  physical  position  between  those  limits. 


TABLE  2.  TRANSFER  PHASE:  FACTOR  VALUES  FOR  THE  HARD,  CENTRAL, 
AND  EASY  TRANSFER  VEHICLE  CONFIGURATIONS 


CONFIGURATION 

FACTORS 

CO 

DL 

TM 

PT 

CG 

A.  Hard  (-) 

100 

.30 

0 

0.0 

.12 

B.  Central  (0) 

62 

.15 

50 

0.0 

.18 

C.  Easy  (♦) 

25 

.00 

100 

0.0 

.24 
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Tracking  Mode  (TM).  Computer-based  control  and  display  manipula¬ 
tions  can  have  effects  on  both  performance  during  training  and  subse¬ 
quent  transfer  of  learning.  For  example,  most  flight  tasks  require 
compensatory  control,  which  appears  to  be  more  than  twice  as  inaccurate 
as  pursuit  tracking  for  fairly  difficult  courses  (Roscoe,  1980).  In  a 
compensatory  tracking  mode  only  the  target  symbol  (Figure  1)  moves  to 
indicate  tracking  error  relative  to  a  fixed  vehicle  index,  usually  at 
the  center  of  the  display.  In  a  pursuit  tracking  mode  both  symbols 
move  to  show  the  absolute  positions  of  the  vehicle  and  the  target 
against  fixed  display  coordinates.  By  modifying  the  training  system  to 
allow  pursuit  control,  it  may  be  possible  that,  because  correct  manual 
inputs  are  elicited  earlier  in  training,  learning  will  be  faster,  and 
transfer  effectiveness  higher.  For  the  present  study,  100$  pursuit 
displays  were  taken  to  be  the  Easy  configuration,  and  100$  compensatory 
displays  the  Hard  configuration.  The  Central  configuration  was  50$ 
pursuit  and  50$  compensatory. 

Prediction  Time  (PT).  Useful  predictors  on  displays  of  the  type 
used  can  be  of  any  order  in  the  Taylor  series.  For  this  study,  a  first- 
order  predictor,  showing  where  the  vehicle  would  be  in  a  few  moments 
with  present  rate  maintained,  was  represented  by  a  small  octagon 
approximating  a  circle  (Figure  1).  In  pursuit  displays  this  symbol 
appeared  as  a  predictor  of  future  vehicle  position,  while  on  compen¬ 
satory  displays  it  appeared  as  a  predictor  of  the  amount  by  which 
present  error  would  be  changed  during  the  prediction  interval.  In 
either  case,  its  position  relative  to  the  vehicle  symbol  represented 
imminent  magnitude  of  error  based  on  current  vehicle  velocity.  Its 
three  values  (-,  0,  and  +)  represented  prediction  times  of  0.0,  0.3, 
and  0.6  second,  respectively. 

Control  Gain  (CC).  Control  or  stick  gain  is  a  modifier  of  the 
output  from  the  stick  to  the  dynamics  of  the  computer-generated  vehi¬ 
cle.  Generally  speaking,  it  may  be  described  as  the  sensitivity  of  the 
vehicle  to  control  input.  For  this  experiment,  a  Central  (0)  value  was 
selected  so  that  full  deflection  of  the  stick  in  a  pure  rate-control 
mode  produced  vehicle  velocity  about  20  percent  greater  than  the  maxi¬ 
mum  forcing-function  velocity.  Thus  the  vehicle  could  always  overtake 
the  target  if  the  operator  applied  full  stick  deflection.  If  this 
Central  value  of  gain  is  regarded  as  one  (or  3/3),  then  the  gain  for 
the  Hard  configuration  was  selected  as  2/3  and  that  for  the  Easy  con¬ 
figuration  as  4/3.  In  other  words,  the  gain  in  the  Hard  condition  was 
66$  and  in  the  Easy  condition  133$  of  the  Central  sensitivity.  The 
actual  unitless  ratio  values  used  for  the  -,  0,  and  conditions  were 
.12,  .18,  and  .24,  respectively. 
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Number  of  Training  Trials  (TT).  Half  of  the  48  transfer  trainees 
"flew"  10  training  trials  (-)  and  the  other  half  30  training  trials  (■*•) 
prior  to  30  transfer  trials  in  one  of  three  transfer  vehicle  configu¬ 
rations.  The  Central  group  flew  20  trials  (0)  with  the  Central  train¬ 
ing  configuration,  followed  by  the  usual  30  transfer  trials  with  the 
Central  transfer  configuration. 

Transfer  Configurations 

Three  different  configurations  of  the  simulated  transfer  criter¬ 
ion  vehicle  were  employed.  Their  actual  compositions  are  shown  in 
Table  2.  One  configuration  was  made  up  of  factor  levels  to  create  a 
Hard  task.  Another  used  the  factor  levels  to  create  an  Easy  task.  A 
Central  configuration  was  of  intermediate  difficulty.  Centrality  wa3 
measured  on  the  physical  scale  of  each  factor  and  did  not  necessarily 
result  in  a  performance  level  midway  between  those  attained  at  the  two 
extremes. 

For  each  transfer  configuration  there  were  16  uniquely  different 
training  configurations.  Participants  who  transferred  to  the  Hard 
configuration  after  training  are  referred  to  as  Group  A,  those  trans¬ 
ferring  to  the  Central  configuration  as  Group  B,  and  those  transferring 
to  the  Easy  configuration  as  Group  C.  A  fourth  group,  D,  also  trans¬ 
ferred  to  the  Central  transfer  configuration,  will  be  described  and  its 
purpose  discussed  in  the  section  on  Data  Collection. 

The  intention  here  was  to  increase  the  generality  of  our  results 
by  creating  an  experimental  space  that  covered  one  extreme  to  the  other 
not  only  in  the  training  phase  but  also  in  the  transfer  phase.  The 
relationships  between  training  and  transfer  levels  covered  extremes  of 
Easy-to-Easy ,  Easy-to-Hard,  Hard-to-Easy ,  and  Hard-to-Hard.  Being 
quantitative  factors,  any  equation  developed  from  the  empirical  data 
would  be  descriptive  (and  thereby  to  some  extent  predictive)  of  any 
point  within  the  limits  of  that  experimental  space.  This  was  the  data 
base  needed  to  discover  training  and  transfer  relationships  across  a 
broad  multifactor  space. 

Participants 

Eighty  right-handed  adult  male  nonpilots  with  self-reported  normal 
or  corrected-to-normal  vision  participated  voluntarily  and  without  pay. 
The  majority  of  these  participants  was  drawn  from  the  Behavioral  Engi¬ 
neering  Laboratory's  participant  roster.  These  people  were  volunteers, 
interested  in  aviation,  recruited  from  the  NMSU  campus  and  local  commu¬ 
nities.  The  remainder  of  the  participants  came  from  the  Department  of 
Psychology's  participant  pool  and  were  fulfilling  a  course  requirement 
by  voluntarily  participating  in  this  particular  experiment. 
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Data  Collection 


It  was  intended  to  sample  performance  at  a  sufficient  number  of 
strategically  located  points  in  the  multifactor  space  to  approximate 
the  complete  training  and  transfer  performance  surfaces  for  the  vari¬ 
ables  studied.  This  goal,  however,  was  subordinated  since  the  data 
collection  effort  had  a  finite  limit  that  did  not  lend  Itself  to  a 
sequential  approach.  The  data  collection  plan  actually  used  was  suffi¬ 
cient  to  estimate  all  main  and  two-factor  interaction  effects  with 
enough  additional  data  to  estimate  how  well  that  limited  model  fit. 
Previous  experience  suggested  that  this  compromise  would  be  adequate. 

Performances  at  49  coordinate  points  were  sampled  in  the  six¬ 
dimensional  training  space  defined  earlier.  Forty-eight  of  these 
points  formed  a  fraction  of  a  3  x  26  factorial  design  divided  into 
three  orthogonal  blocks  represented  by  Groups  A,  B,  and  C,  who  trans¬ 
ferred,  respectively,  to  the  Hard,  Central,  and  Easy  transfer  vehicle 
configurations.  From  the  16  points  within  each  block,  all  main  effects 
could  be  estimated  independently  from  one  another  and  from  all  two- 
factor  interactions  with  the  interactions  aliased  (i.e.  confounded  with 
each  other)  in  strings.  With  data  from  the  three  blocks  combined,  all 
main  effects  and  all  two-factor  interactions  could  be  isolated  from 
one  another,  although  not  from  higher-order  interactions  (Connor  and 
Young,  1961,  p.  16). 

The  49th  point  was  located  at  the  center  of  the  fractional  fac¬ 
torial  design  and  was  replicated  eight  times.  Thus,  eight  partici¬ 
pants,  designated  Group  D,  were  trained  at  this  central  position  and 
subsequently  transferred  to  the  Central  transfer  configuration  (the 
same  as  that  for  Group  B).  These  centerpoint  data  provided  an  estimate 
of  experimental  error  variance  and,  when  combined  with  the  other  data, 
allowed  a  test  of  whether  or  not  a  linear  second-order  model  was 
adequate.* 

The  coordinates  for  this  data  collection  plan  are  given  in  Appen¬ 
dix  A.  A  different  subject  was  assigned  to  each  of  the  48  fractional 
factorial  points  and  the  eight  centerpoints,  making  a  total  of  56 
transfer  participants  who  received  the  instructions  in  Appendix  B  and 
were  then  trained  and  tested  in  the  following  sequence: 


•These  centerpoints  do  not  affect  the  estimates  of  the  effects  of 
the  two-level  factors.  A  slightly  better  design  might  have  been  to 
include  two  or  three  Central  training  conditions  within  each  of 
groups  A,  B,  and  C. 


1.  Matching  trials:  Each  participant  was  given  three 
trials  in  the  Central  training  configuration.  The 
median  of  these  three  scores  was  used  to  adjust 
training  and  transfer  scores  for  individual  differ¬ 
ences  in  initial  ability. 

2.  Masking  trials:  Each  participant  were  then  given 
two  trials  using  his  individually  assigned  training 
configuration.  The  purpose  of  this  was  to  allow  any 
immediate  carryover  effects  from  the  matching  condi¬ 
tion  to  the  training  condition  to  dissipate. 

3.  Training  trials:  Each  participant  was  given  10,  20,  or 
30  trials  on  his  specific  training  vehicle  configuration, 
the  number  depending  on  his  coordinate  position  in  the 
six-dimensional  training  space. 

Total  instructional  and  "flying"  time  for  these  first  three  steps 
of  the  data  collection  ranged  from  approximately  25  to  U5  minutes 
depending  on  the  number  of  training  trials  involved. 

4.  Transfer  trials:  Following  a  30-minute  rest  period  away 
from  the  experimental  room,  each  participant  was  brought 
back  and  tested  for  30  trials  on  one  of  the  three  transfer 
vehicle  configurations. 

Three  additional  groups  of  eight  participants  each  also  received 
the  instructions  in  Appendix  B  and  were  tested  to  provide  "control" 
performances  on  the  three  transfer  vehicle  configurations.  Each  group 
was  trained  on  a  different  transfer  configuration  without  prior  train¬ 
ing  on  any  other  configuration  except  for  the  matching  trials.  As  with 
the  transfer  groups,  each  control  participant  was  given  the  three 
matching  trials  at  the  center  of  the  training  space  and  the  two  masking 
trials  with  his  particular  training  vehicle  configuration  prior  to  the 
30  scored  training  trials. 
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SECTION  IV 

MAXIMIZING  AND  EVALUATING  DATA  QUALITY 

As  experimental  designs  become  more  complex  and  capable  of  gener¬ 
ating  large  quantities  of  information  from  relatively  few  data  points, 
a  necessary  first  step  in  the  analysis  is  to  examine  the  quality  of  the 
data.  This  i3  imperative  to  ensure  its  valid  interpretation.  The  data 
in  this  study  were  examined  (a)  to  decide  on  proper  analyses,  and  (b) 
to  detect  characteristics  of  the  data  that  might  distort  interpreta¬ 
tion.  Rather  than  accept  the  numbers  uncritically,  the  following 
questions  were  asked: 

1.  In  what  form  are  the  data  most  suitable  for  analysis? 

2.  Were  the  assumptions  of  control  and  balance  in  the 
experimental  design  actually  met? 

3-  Are  the  data  distorted  by  any  experimental  procedure? 

4.  Do  the  data  behave  a3  expected? 

5.  Are  there  internal  inconsistencies  in  the  data? 

6.  Are  observed  outliers  meaningful  or  consequences  of 
poor  data  collection  techniques? 

Analyses  were  performed  to  answer  these  and  similar  questions.  The 
more  important  conclusions  are  listed  below: 

1 .  To  provide  a  more  normal  distribution  of  the  performance 
data,  logarithmic  transformations  were  made  of  the  orig¬ 
inal  RMS  error  scores.  All  further  discussion  of  per¬ 
formance  scores  will  be  in  terms  of  the  log  RMS  values. 

2.  In  combining  the  results  from  several  trials  into  a 
single  measure,  the  median  value  (rather  than  the  mean) 
was  used. 

3.  The  seven  groups  (A,  B,  C,  D,  and  three  controls)  were 
well-matched  in  terms  of  the  means  and  variances  of 
their  matching  scores. 

4.  Although  the  data  were  collected  by  three  experimenters 
over  a  period  of  several  weeks  and  at  different  times 
of  day  and  evening,  there  was  no  evidence  that  these 
sources  of  variance  biased  the  data  to  any  practical 
extent. 
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5.  The  amount  of  learning  during  training  was  not  large, 
but  the  overall  curves  showed  expected  patterns. 

Individual  curves,  however,  were  characterized  by 
large  variations,  both  within  and  between  configura¬ 
tions. 

6.  The  assumptions  made  in  selecting  the  Hard  and  Easy 
levels  for  the  factors  and  the  transfer  configura¬ 
tions  were  validated.  Only  Prediction  Time  did  not 
conform  to  expectations. 

Supporting  data  regarding  these  conclusions  are  given  in  Appendix  C. 
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SECTION  V 

TRAINING  AND  TRANSFER:  SURFACES  AND  EFFECTS 

The  data  were  analyzed  to  obtain  information  regarding  the  train¬ 
ing  and  transfer  surfaces  as  well  as  the  individual  effects.  To  de¬ 
scribe  the  training  and  transfer  performance  surfaces,  data  from  the  56 
participants  in  the  four  experimental  groups  were  used.  When  transfer 
effects  are  considered,  these  values  were  obtained  by  adjusting  each 
individual’s  transfer  performance  score  by  the  mean  performance  level 
of  the  control  group  tested  with  the  same  transfer  vehicle  configura¬ 
tion. 

Individual  Initial  'Abilities 


Prior  to  training,  each  of  the  80  participants  was  tested  for 
three  trials  on  the  same  (Central)  system  configuration.  The  median 
of  these  three  trials  was  used  to  represent  each  participant's  initial 
ability  or  skill  level  and  is  referred  to  as  his  matching  score.  The 
mean  of  the  matching  scores  for  the  80  participants  was  1689.44,  with 
a  standard  deviation  of  82.15  and  a  standard  error  of  the  mean  of  9.18. 
The  smallest  value  was  1520,  approximately  two  standard  deviations 
below  the  mean;  the  largest  value  was  1912,  approximately  two  and  three- 
quarters  standard  deviations  above  the  mean. 

Differences  among  the  individual's  initial  abilities,  as  measured 
by  the  matching  scores,  were  partialed  out  of  all  training  and  transfer 
scores.*  All  subsequent  references  to  training  performances,  transfer 
performances,  and  transfer  effects  are  based  on  the  adjusted  values 
from  which  that  portion  of  a  score  attributable  to  initial  individual 
ability  has  been  partialled  out. 

Interpreting  Signs 

Throughout  this  paper,  plus  and  minus  signs  are  used  to  represent 
the  coded  levels  of  the  factors  on  the  input  side  of  the  experiment  and 
the  signs  of  the  factor  effects  (twice  the  coefficients  of  the  regres¬ 
sion  equations)  and  transfer  effects  scores  on  the  output  side.  To 
facilitate  the  interpretation  of  these  signs,  Table  3  is  provided  here, 
to  be  used  when  needed  to  understand  the  results  throughout  the 
remainder  of  this  paper. 


*Caution:  With  designs  in  which  conditions  and  subjects  are  con¬ 
founded,  it  is  important  that  acorea  of  the  matching  task  clearly 
reflect  individual  differences  in  ability.  Prevalidation  of  the 
matching  task  is  paramount. 


'ABLE  3-  LEGEND  FOR  INTERPRETING  SIGNS 


INPUTS 

Coded  levels  of  equipment/training  factors 

-1  Preexperiment,  assumed  Hard  (High  log  RMS  error) 

♦1  Preexperiment,  assumed  Easy  (Low  log  RMS  error) 

Coded  levels  of  similarity  variables: 

Direction  (Difficulty) 

-1  Hard  to  Easy 

0  No  change 

♦1  Easy  to  Hard 

Distance  (Fidelity) 

-1  Same  position  (0) 

0  Moved  1  position 

+1  Moved  2  positions 

OUTPUTS 

Factor  Coefficients  (Coeff.  x  2  =  Mean  difference  =  Effect) 
for  training  and  transfer  scores 

-  Hard  Level  had  larger  RMS  error  than  Easy  Level 
+  Easy  Level  had  larger  RMS  error  than  Hard  Level 

Transfer  effects  scores  (observed  minus  control  scores) 

-  Positive  Transfer  (Control  error  higher  than  observed) 

+  Negative  Transfer  (Control  error  lower  than  observed) 

Similarity  Coefficients  (Coeff.  x  2  =  Mean  difference  =  Effect) 

Direction  ^Difficulty) 

-  Hard-to-Easy  error  greater  than  Easy-to-Hard, 
or  Hard-to-Easy  had  poorer  transfer 

+  Easy-to-Hard  error  greater  than  Hard-to-Easy, 
or  Hard-to-Easy  had  better  transfer 

Distance  (Fidelity) 

-  Shorter  distance  had  larger  error  than  Longer 
distance,  or  Shorter  had  poorer  transfer 

*  Longer  distance  had  larger  error  than  Shorter 
distance,  or  Longer  had  poorer  transfer 
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Training  Data 

The  median  scores  for  the  last  five  training  trials,  regardless  of 
the  total  number  of  trials,  were  analyzed  for  the  training  surface 
characteristics  and  for  the  effects  of  the  individual  factors  and  their 
interactions.  These  analyses  were  applied  to  the  combined  set  of  56 
scores  for  Experimental  Groups  A,  B,  C,  and  D  and  separately  to  each  of 
the  three  sets  of  16  scores  for  Groups  A,  B,  and  C  who  subsequently 
transferred  to  the  Hard,  Central,  and  Easy  vehicle  configurations. 
Summaries  of  the  analyses  for  surface  characteristics  of  the  training 
performance -data  are  shown  in  Tables  4,  6,  8,  and  10  and  for  the  factor 
effects  in  Table  12. 

Transfer  Data 


Median  performances  on  the  first  five  trials  with  the  respective 
transfer  vehicle  configurations  were  used  for  all  participants,  both 
those  trained  and  the  control  groups.  Transfer  performance  scores  for 
the  56  trained  operators  were  analyzed  in  the  same  manner  as  the  train¬ 
ing  data.  To  estimate  the  transfer  effect  scores,  group  mean  perform¬ 
ance  scores  for  each  of  the  three  control  groups  were  subtracted  from 
the  corresponding  individual  transfer  performance  scores  of  the 
experimental  groups.  Thus,  for  individual  measures,  positive  transfer 
savings  are  indicated  by  negative  difference  scores  (large  control 
scores  subtracted  from  smaller  transfer  scores),  and  conversely 
negative  transfer  effects  are  indicated  by  positive  difference  scores. 

Summaries  of  the  analyses  of  surface  characteristics  of  the  trans¬ 
fer  performance  data  are  shown  in  Tables  5,  7,  9,  and  11,  those  for  the 
factor  effects  in  Table  13*  The  analysis  of  the  transfer  effects  is 
not  shown  but  will  be  covered  in  the  discussion  of  Table  13.  See 
Appendix  D  regarding  the  percent  transfer  analysis. 

Training  and  Transfer  Surface  Characteristics 

When  Tables  4  and  5  are  examined  and  compared,  the  following 
generalizations  can  be  drawn: 

1.  Both  training  and  transfer  surfaces  can  be  approximated 
predominantly  by  a  combination  of  first-order  effects. 

2.  The  training  surface  is  somewhat  more  complex  than  the 
transfer  surface,  being  influenced  more  by  two- factor 
interactions  and  showing  a  somewhat  greater  lack  of  fit. 

3.  Evidence  of  curvature  is  slight  in  both  surfaces. 

4.  Experimental  factors  have  a  major  influence  on  perform¬ 
ance  during  training  but  trivial  effects  on  transfer 

in  the  presence  of  the  large  performance  differences 
imposed  by  the  three  transfer  vehicle  configurations. 
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TABLE  4.  ANALYSIS  OF  TRAINING  PERFORMANCE  SURFACE: 
GROUPS  A,  3,  C,  AND  D 


Source  of 

Variance 

df 

Mean 

Squares 

F 

'2 

Proportion 
of  Variance 

Regression: 

35 

71531 

79.0 

.000* 

.94 

First-order  terms 

8 

261126 

283.5 

.000 

.79 

2  Levels 

6 

332868 

367.8 

.000 

.75 

3  Levels 

2 

45902 

50.7 

.000 

.03 

Second-order  terms 

27 

15354 

16.9 

.000 

.1b 

2  FI  (2  x  2  levels) 

15 

17201 

19.0 

.000 

.  10 

2  FI  (2  x  3  levels) 

12 

13406 

14.4 

.001 

.06 

Residual: 

20 

7461 

.06 

Lack  of  Fit 

12 

4742 

5.2 

.025 

.  02 

Curvature 

1 

85981 

95.0 

.000 

.03 

Error 

7 

905 

.00 

TABLE  5.  ANALYSIS  OF 

TRANSFER  1 

PERFORMANCE 

SURFACE: 

GROUPS  A,  B 

,  C,  AND  D 

Source  of 

Mean 

Proportion 

Variance 

df 

Squares 

F 

c2 

of  Variance 

Regression: 

35 

190125 

43.7 

•  000» 

.97 

First-order  terms 

8 

744636 

163.3 

.000 

.67 

2  Levels 

6 

26l8l 

5.7 

.025 

.02 

3  Levels 

2 

2900002 

636.1 

.000 

.34 

Second-order  terms 

27 

25826 

5.7 

.025 

.10 

2  FI  (2  x  2  levels) 

15 

24794 

5.4 

.025 

.05 

2  FI  (2  x  3  levels) 

12 

27116 

5.9 

.025 

.05 

Residual : 

20 

11267 

.03 

Lack  of  Fit 

12 

5558 

1.3 

.10 

.00 

Curvature 

1 

128170 

29.4 

.010 

.03 

Error 

7 

4354 

.00 

*£  =  .000  is  <  .0005. 
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TABLE  6.  ANALYSIS  OF  TRAINING  PERFORMANCE  SURFACE:  GROUP  A 


Source  of 

Variance 

df 

Mean 

Squares  F 

<£ 

Proportion 
of  Variance 

First-order  terms 

6 

13M75  8.6 

.005 

.85 

Residual 

9 

156635 

.15 

Second-order  terms 
(aliased) 

7 

18085  2.6 

.10 

.13 

Remainder 

2 

7061 

.01 

TABLE  7. 

ANALYSIS  OF 
HARD  VEHICLE 

TRANSFER  PERFORMANCE 
CONFIGURATION  (GROUP 

SURFACE: 

A) 

Source  of 

Variance 

df 

Mean 

Squares  F 

<£ 

Proportion 
of  Variance 

First-order  terms  6  1030*1  0.7  .10  .31 

Residual  9  15633  .69 

Second-order  terms  7  17500  1.9  .10  .60 

(aliased) 


Remainder 


2 


9101 
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TABLE  3.  ANALYSIS  OF  TRAINING  PERFORMANCE  SURFACE:  GROUP  3  i 

1 


Source  of 

Variance 

df 

Mean 

Squares 

F 

Proportion 
of  Variance 

First-order  terms 

6 

106397 

9.6 

.005 

.87 

Residual 

9 

1 1085 

.13 

Second-order  terms 

7 

1 1 162 

1.0 

.  10 

.  1 1 

(aliased) 

Remainder 

2 

I06l6 

.03 

TABLE  9. 

ANALYSIS 

CENTRAL 

OF  TRANSFER  PERFORMANCE  SURFACE: 
VEHICLE  CONFIGURATION  (GROUP  3) 

Source  of 

Mean 

Proportion 

Variance 

df 

Squares  F  -  0 

of  Variance 

First-order  terms 

6 

55631  3.0  .06 

Residual 

9 

D453 

-v 

.  zc 

Second-order  terms 
(aliased) 

7 

4913  1.7  .1C 

.27 

Remainder 

2 

2345 

.04 

34 


NAVTRAEQUIPCEN  78-C-0060-6 


I 

t 


TABLE  10.  ANALYSIS  OF  TRAINING  PERFORMANCE  SURFACE:  GROUP  C 


Source  of 

Variance 

df 

Mean 

Squares 

F 

Proportion 
of  Variance 

First-order  terms 

6 

117*85 

11.1 

.001 

.88 

Residual 

9 

10616 

.12 

Second-order  terras 
(aliased) 

7 

10716 

1.0 

.10 

.09 

Remainder 

2 

10268 

.03 

TABLE  11. 

ANALYSIS  OF 
EASY  VEHICLE 

TRANSFER  PERFORMANCE 
CONFIGURATION  (GROUP 

SURFACE: 

C) 

Source  of 

Variance 

df 

Mean 

Squares 

F 

<2 

Proportion 
of  Variance 

First-order  terms 

6 

55681 

1.3 

.10 

.47 

Residual 

9 

42623 

.53 

Second-order  terms 
(aliased) 

7 

42168 

1.0 

.10 

.41 

Remainder 

2 

44216 

.12 
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TABLE  12.  ANALYSIS  OF  TRAINING  PERFORMANCE  EFFECTS: 
GROUPS  A,  B,  C,  AND  D 


Source  of 

Effect* 

Regression 

Coefficient** 

Standard 

Error 

Standardized 

Coefficient 

£ 

INTERCEPT 

1710.9 

Control  Order 

-123.5 

12.5 

-.53 

.000 

Display  Lag 

-157.9 

12.5 

-.67 

.000 

Tracking  Mode 

-36.5 

12.5 

-.16 

.008 

Prediction  Time 

2.7 

12.5 

.01 

.831 

Control  Gain 

-2.0 

12.5 

-.01 

.872 

Training  Trials 

-8.8 

12.5 

-.04 

.483 

TVC,  Linear  (XL) 

-47.2 

15.3 

-.16 

.006 

TVC ,  Quadratic  (XQ) 

16.6 

10.0 

.09 

.113 

CO  x  DL 

-38.7 

15.3 

-.17 

.020 

CO  x  TM 

-9.1 

13.2 

-.04 

.500 

CO  x  PT 

13.1 

13.2 

.06 

.335 

CO  x  CG 

-4.0 

13.2 

-.02 

.764 

CO  x  TT 

6.6 

13-2 

.03 

.622 

CO  x  XL 

3.5 

15.3 

.01 

.820 

CO  x  XQ 

11.5 

8.8 

.07 

.206 

DL  x  TM 

-29.9 

13-2 

-.13 

.035 

DL  x  PT 

39.9 

13.2 

.17 

.007 

DL  x  CG 

-12.6 

13*2 

-.05 

.353 

DL  x  TT 

6.2 

13.2 

.03 

.644 

DL  x  XL 

-3.6 

15.3 

-.01 

.814 

DL  x  XQ 

-19.6 

8.8 

-.12 

.038 

TM  x  PT 

19.6 

13.2 

.08 

.155 

TM  x  CG 

25.2 

15.3 

.11 

.115 

TM  x  TT 

-10.2 

13.2 

-.04 

.452 

TM  x  XL 

-29.9 

15.3 

-.10 

.065 

TM  x  XQ 

5.9 

8.8 

.04 

.509 

PT  x  CG 

-19.9 

13-2 

-.09 

.148 

PT  x  TT 

22.0 

15.3 

.09 

.  165 

PT  x  XL 

6.8 

15.3 

.02 

.662 

PT  x  XQ 

-14.3 

8.8 

-.09 

.119 

CG  x  TT 

-9.6 

13.2 

-.04 

.477 

CG  x  XL 

-2.2 

15.3 

-.01 

.887 

CG  x  XQ 

10.4 

8.8 

.06 

.250 

TT  x  XL 

-20.3 

15.3 

-.07 

.  198 

TT  x  XQ 

-17.4 

8.8 

-.11 

.063 

•Sources  that  were  included  in  ah  equation  from  a  stepwise  regression 
analysis  with  F  =  4.00  to  enter  and  exit  are  underlined. 

••Two  times  the  Regression  Coefficient  equals  the  mean  difference 
between  ♦  and  -  levels.  For  interactions,  the  mean  difference  is 
between  (+♦  and  — )  and  (♦-  and  — ♦•). 
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TABLE  13:  ANALYSIS  OF  TRANSFER  PERFORMANCE  EFFECTS: 
GROUPS  A,  B,  C,  AND  D 


Source  of 

Effect* 

Regression 

Coefficient** 

Standard 

Error 

Standardized 

Coefficient 

2 

INTERCEPT 

Control  Order 

1688.2  (-11.1) 
-14.5 

15.3 

-.04 

.355 

Display  Lag 

-32.4 

15.3 

-.09 

.047 

Tracking  Mode 

31.7 

15.3 

.08 

.052 

Prediction  Time 

6.5 

15.3 

.02 

.678 

Control  Gain 

31.1 

15.3 

.08 

.056 

Training  Trials 

0.5 

15.3 

.00 

.975 

TVC,  Linear  (XL) 

-408.8  (-70.3) 

18.8 

-.88 

.000 

TVC,  Quadratic  (XQ) 

78.0  (+31.7) 

12.3 

.26 

.000 

CO  x  DL 

4.2 

18.8 

.01 

.825 

CO  x  TM 

-20.9 

16.3 

-.06 

.213 

CO  x  PT 

22.4 

16.3 

.06 

.184 

CO  x  CG 

13.2 

16.3 

.04 

.427 

CO  x  TT 

11.7 

16.3 

.03 

.480 

CO  x  XL 

-27.6 

18.8 

-.06 

.157 

CO  x  XQ 

10.4 

10.8 

.04 

.350 

DL  x  TM 

-18.2 

16.3 

-.05 

.277 

DL  x  PT 

27.6 

16.3 

.07 

.105 

DL  x  CG 

2.2 

16.3 

.01 

•  895 

DL  x  TT 

35.7 

16.3 

.09 

.040 

DL  x  XL 

-10.3 

18.8 

-.02 

.588 

DL  x  XQ 

7.0 

10.8 

.03 

.524 

TM  x  PT 

-39.1 

16.3 

-.10 

.026 

TM  x  CG 

-5.3 

18.8 

-.01 

.782 

TM  x  TT 

18.8 

16.3 

.05 

.260 

TM  x  XL 

-58.2 

18.8 

-.13 

.006 

TM  x  XQ 

25.5 

10.8 

.09 

.029 

PT  x  CG 

27.4 

16.3 

.07 

.107 

PT  x  TT 

-20.2 

18.8 

-.05 

.296 

PT  x  XL 

22.4 

18.8 

.05 

.246 

PT  x  XQ 

-14.9 

10.8 

-.06 

.185 

CG  x  TT 

-46.7 

16.3 

-.12 

.009 

CG  x  XL 

-20.0 

18.8 

-.04 

..300 

CG  x  XQ 

24.7 

10.8 

.09 

.034 

TT  x  XL 

10.0 

18.8 

.02 

.599 

TT  x  XQ 

-.7 

10.8 

-.00 

.952 

•Sources  that  were  included  in  an  equation  from  a  stepwise  regression 
analysis  with  F  =  4.00  to  enter  and  exit  are  underlined. 

••Two  tines  the  Regression  Coefficient  equals  the  mean  difference  between 
♦  and  -  levels.  For  interactions,  the  mean  difference  is  between  (+♦ 
and  — )  and  (♦-  and  -+) .  Coefficients  in  parentheses  are  those  of  the 
transfer  effects  that  were  different  from  transfer  performance  effects 
(see  page  3D* 
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When  the  data  for  each  group  associated  with  a  transfer  vehicle  con¬ 
figuration  are  examined  separately  (Tables  6,  7,  3,  9,  10,  11)  the 
following  generalizations  can  be  made: 


1.  The  combined  first-order  effects  of  the  experimental  factors 
account  for  most  of  the  variation  in  performance  curing 
training  for  each  group.  (Mote:  Transfer  vehicle  configu¬ 
rations  were  not  a  source  of  variance  during  training.) 

2.  No  clear  pattern  can  be  seen  in  the  results  of  the  transfer 
data  across  the  three  configurations.  Effects  are  marginal 
or  trivial.  Compared  to  the  training  surface,  the  transfer 
surface  is  relatively  flat. 


Interpretation  of  Training  ar.d  Transfer  Effects 

The  contributions  of  the  individual  sources  of  variance,  namely  35 
main  and  two-factor  interaction  effects,  during  training  and  transfer 
are  shown  in  Column  2  of  Tables  12  and  13,  respectively.*  Each  coeffi- 
v  ant  shows  the  change  in  performance  per  unit  change  in  the  corre¬ 
sponding  source  of  variance.  Since  performance  is  measured  in  log  RMS 
error,  a  coefficient  with  a  negative  sign  indicates  that  performance 
was  poorer  on  the  (-1)  level,  designated  Hard,  of  the  factor  associated 
with  the  coefficient.  The  mean  difference  between  the  easy  and  hard 
levels  of  any  factor  can  be  obtained  by  multiplying  its  coefficient  by 
two.  For  two-factor  interactions,  the  difference  is  between  the  means 
of  those  values  in  which  both  factors  are  of  the  same  sign  {■*■+  and  — ) 
and  those  in  which  they  have  different  signs  (+-  and  -+). 

Standardized  regression  coefficients  (SRCs)  in  Column  are  calcu¬ 
lated  from  scores  normalized  in  units  of  their  own  standard  deviations 
about  the  mean.  SRCs  squared  approximate  the  proportion  of  total 
variance  accounted  for  by  each  source  after  other  sources  have  been 
partialled  out.  The  probability  that  an  individual  coefficient  of  the 
size  shown  in  Column  2  would  occur  by  chance  is  provided  (for  a  two- 
tailed  £-test)  in  Column  5.  A  £  value  of  ”.000”  is  actually  any  value 
smaller  than  .001.  Each  source  accounts  for  one  degree  of  freedom;  for 
these  calculations,  there  were  20  degrees  of  freedom  in  the  error  term. 


•The  three-level  factor,  Transfer  Vehicle  Configuration  (TVC),  was 
divided  into  two  orthogonal  components,  namely  the  linear  and  quadratic 
trend  effects  (XL  and  XQ,  respectively)  with  one  degree  of  freedom 
each.  XL  provides  a  comparison  of  the  two  end  levels,  and  XQ  compares 
the  mean  of  the  two  end  levels  with  the  middle  level.  These  interact 
with  the  other  factors  in  the  conventional  manner. 
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The  underlined  Sources  of  Effects  in  Tables  12  and  13  indicate 
those  terms  that  emerge  from  a  stepwise  regression  analysis  in  which 
the  criterion  for  entry  and  exit  is  an  F- value  of  4.00.  The  coeffi¬ 
cients  for  these  terms  in  the  stepwise  analysis  will  be  the  same  as 
those  in  the  full  regression  analysis  except  when  one  source  overlaps 
(interacts  with)  others.  In  the  full  analysis,  the  reported  coeffi¬ 
cients  are  corrected  for  overlap.  In  the  stepwise  analysis,  no  cor¬ 
rection  is  made  unless  both  interactions  in  an  overlapping  pair  are 
brought  into  the  equation.  Thus,  the  differences  that  may  exist  for 
some  terms  are  slight  and  not  relevant  to  this  discussion. 

If  we  examine  Tables  12  and  1 3‘»  certain  results  stand  out  clearly: 

1.  Factors  CO  (Control  Order),  and  DL  (Display  Lag),  show 
relatively  large  effects  during  training  but  small  or 
marginal  effects  during  transfer. 

2.  In  transfer,  the  three  Transfer  Vehicle  Configurations 
(Factors  XL  and  XQ)  overwhelm  all  other  sources  of 
variance  (this  was  as  intended,  of  course),  with  the  next 
largest  sources  being  interactions  (TM  x  XL,  CG  x  TT,  TM 

x  PT)  rather  than  main  effects,  and  these  contributed  only 
10  or  12  percent  as  much  variance  as  the  linear  Transfer 
Vehicle  Configurations  effect  (XL). 

3.  It  would  appear  that  fewer  than  ten  of  the  35  isolated 
sources  of  variance  in  either  group  had  a  critical  effect 
on  performance,  quite  in  line  with  the  Principle  of  Mal¬ 
distribution  (Simon,  1973,  1977),  a  fundamental  assumption 
in  economical  multifactor  research. 

Selecting  which  effects  are  truly  critical  is  difficult  in  this 
study  because  there  are  no  real-world  criteria  to  use.  Ordinarily  one 
would  look  at  the  mean  differences  first  to  see  which  effects  approach 
operationally  critical  values.  Then  one  would  examine  the  proportions 
of  variance  accounted  for  and  tests  of  statistical  significant i.  These 
three  criteria  are  not  necessarily  consistent  and  require  careful 
interpretation.  Using  the  £-values  in  Column  5  as  the  basis  for  selec¬ 
tion  requires  consideration  of  the  number  of  effects  being  examined,  in 
this  case  35.  Since  proper  selection  of  critical  factors  depends  in 
part  on  real  system  requirements,  there  are  no  good  rules  for  deciding 
in  this  artificial  situation. 

In  Tables  14  and  15,  the  results  from  stepwise  regression  analyses 
of  Groups  A,  B,  and  C  individually  associated  with  the  three  vehicle  con¬ 
figurations  during  transfer  are  shown  for  both  the  training  and  transfer 
data.  In  all  cases,  an  F  of  4.00  was  used  as  the  criterion  for  entry  to 
or  exit  from  the  equation.  While  these  analyses  are  based  on  only  16 
observations  each,  they  allow  an  examination  of  the  results  within  trans¬ 
fer  configurations  rather  than  between  transfer  configurations  and  thereby 
eliminate  comparison  of  interactions  of  other  factors  with  XL  and  XQ. 


TABLE  14.  STEPWISE  REGRESSION  ANALYSIS  OF  TRAINING  PERFORMANCE 
WITH  F  =  4.00  TO  ENTER  AND  EXIT 


Group  A 


Equation:  1774.67. 


( • 27 ) •  (.60)  (.03) 

115.53  CO  -  173-86  DL  40.83  (CO  x  TT  +  DL  x  PT) 


Adjusted  R2  =  .880 
S.E.  of  Coefficient  =  19.97 


F  =  37.80  (3 r\2),  2  <  -001 
S.E.  of  Estimate  =  78.90 


Group  B 


Equation: 


(.46)  (.30) 

1725.73  -  146.61  CO  -  118.64  DL 


Adjusted  jf  =  .732 
S.E.  of  Coefficient  =  28.76 


F  =  21.50  (2/13),  £  <  .001 
S.E.  of  Estimate  =  115.04 


Group  C 

(.20)  (.55) 

Equation:  1680.27  -  108.50  CO  -  181 . 14  DL 

Adjusted  R2  =  .715  F  =  19.80  (2/13),  £  <  .001 

S.E.  of  Coefficient  =  33-56  S.E.  of  Estimate  =  134.23 


•Numbers  in  parentheses  above  factors  indicate  proportions  of  total 
variance  accounted  for. 
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TABLE  15.  STEPWISE  REGRESSION  ANALYSIS  OF  TRANSFER  PERFORMANCE 
WITH  F  s  4.00  TO  ENTER  AND  EXIT 

Hard  Vehicle  Configuration  (Group  A) 

( .305* 

Equation:  2174.91+  115.35  TM 

Adjusted  R2  =  .246  F  =  5.90  (1/14),  £  <  .05 

S.E.  of  Coefficient  =  47.47  S.E.  of  Estimate  =  189.89 


Central  Vehicle  Configuration  (Group  B) 

(.27) 

Equation:  1591.14  -  46.47  DL 

Adjusted  R2  =  .22  F  =  5.24  (1/14),  £  <  .05 

S.E.  of  Coefficient  =  20.30  S.E.  of  Estimate  =  81.21 

Easy  Vehicle  Configuration  (Group  C) 

(.31) 

Equation:  1357.34  +  62.80  (CO  x  DL  +  DL  x  TT) 

Adjusted  R2  =  .26  F  =  6.33  0/14),  £  <  .025 

S.E.  of  Coefficient  =  24.95  S.E.  of  Estimate  =  99.81 


•Numbers  in  parentheses  above  factors  indicate  p.  oportions  of  total 
variance  accounted  for. 
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Important  sources  of  variance.  Control  Order  and  Display  Lag  (CO 
and  DL)  have  strong  effects  on  the  training  performance  of  each  group. 
These  two  factors  accounted  for  more  than  705  of  the  variance  in  each 
analysis.  With  the  transfer  data,  the  picture  is  different.  No  more 
than  a  single  factor  or  interaction  string  meets  the  criterion  for 
entry  into  the  transfer  equation  for  any  group,  and  entries  are  not 
consistent  from  group  to  group.  Terms  admitted  to  each  of  these  equa¬ 
tions  account  for  only  about  305  of  the  total  variance.  There  i3  not 
sufficient  information  to  decide  why  a  particular  factor  affected  a 
particular  transfer  configuration,  nor  to  determine  what  caused  the 
unexplained  variability. 

Transfer  effects.  The  analyses  of  transfer  effects  scores  yield 
the  same  coefficients  as  those  for  the  transfer  performance  scores, 
shown  in  Table  13,  with  three  exceptions:  the  intercept,  XL,  and  XQ. 
These  three  values  are  listed  in  parentheses  on  the  proper  lines  in 
Column  2.  The  reason  for  this  is  that  when  the  average  performance  of 
the  appropriate  control  group  is  removed  from  each  transfer  performance 
score,  the  differences  among  transfer  vehicle  configuration  levels  (XL 
and  XQ)  are  diminished  accordingly.  However,  the  resulting  "transfer 
effect"  scores  will  be  affected  in  exactly  the  3ame  way  by  the  other 
experimental  factors  as  the  transfer  performance  scores  are,  and  by 
the  3ame  amount.  See  Appendix  D  regarding  percent  transfer  scores. 

Relations  between  training  and  transfer.  To  facilitate  examina¬ 
tion  of  factor  effects  across  training  and  transfer  boundaries,  the  48 
performance  measures  of  groups  A,  B,  and  C  from  both  phases  were  com¬ 
bined.  The  96  scores  were  subjected  to  a  stepwise  regression  analysis 
(F  =  4.00  to  enter  and  exit)  with  all  the  main  effects  and  two-factor 
interactions  also  interacting  with  a  new  factor,  termed  experiment 
Phase  (P),  which  has  training  (-1)  and  transfer  (+1)  as  its  two  levels. 
This  enabled  the  effects  of  71  sources  of  variance  to  be  isolated  (with 
Transfer  Vehicle  Configuration  divided  into  linear  and  quadratic 
sources).  There  were  nine  main  effects,  35  two-factor  interactions, 
and  27  three-factor  interactions  to  be  considered. 

Based  on  the  criterion  indicated  earlier,  17  of  the  71  possible 
terms  were  used  in  the  equation.  In  combination,  these  accounted  for 
91%  of  the  total  variance  of  the  96  scores.  These  terms  are  listed  in 
order  of  their  strength  of  effect  on  performance  in  Table  16  where 
component  XL  and  XQ  terms  have  been  combined  into  the  single  Transfer 
Vehicle  Configuration  variable,  X  (with  two  degrees  of  freedom).  The 
more  interesting  of  these  have  been  plotted  in  Figure  2  and  will  be 
discussed  below.  Those  of  greatest  interest  are  the  two-  and  three- 
factor  interactions  that  involve  both  training  and  transfer  phases. 
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TABLE  16.  RESULTS  FROM  STEPWISE 
COMBINED  TRAINING  AND 
WITH  F  =  4.00  TO  ENTER 

REGRESSION  ANALYSIS  OF 

TRANSFER  PERFORMANCE  DATA 

AND  EXIT  (N  =  96) 

Source  of 

Effect 

Regression  Proportion  of 

Coefficient  Total  Variance 

Linear  (L) 

T  V  Configuration  (X) 

Quadratic  (Q) 

-227\ 

♦29-'' 

.376 

L 

Phase*  x  X 

Q 

•’V 

+29^^ 

.242 

Display  Lag 

-95 

.094 

Control  Order 

-69 

.050 

Display  Lag  x  Phase 

♦63 

.04 1 

Control  Order  x  Phase 

♦55 

.031 

L 

Tracking  Mode  x  X 

Q 

-‘“'V 

.032 

Display  Lag  x  Prediction  Time 

+37 

.014 

Tracking  Mode  x  Phase 

♦34 

.012 

Display  Lag  x  Training  Trials 

♦27 

.007 

Control  Gain  x  Training  Time 

-25 

.006 

Tracking  Mode  x  Prediction  Time  x  Phase 

-23 

.006 

Display  Lag  x  Tracking  Mode 

-23 

.005 

Control  Gain  x  XQ 

♦18 

.005 

•Phase  refers  to  Training  Trials  vs.  Transfer  Trials. 


43 


1000  log  10  RMS  ERROR 


Figure  2.  Critical  interactions  between  equipment/training  factors 
and  phases. 


The  three  largest  sources  of  variance  in  this  analysis  are  still 
the  main  effects  of  Factors  X  (Transfer  Vehicle  Configuration),  CO 
(Control  Order),  and  DL  (Display  Lag),  but  this  observation  is  simplis¬ 
tic  and  misleading.  This  is  so  because  all  three  of  these  sources  also 
show  strong  interactions  with  the  phase  of  training.  The  plots  of  X  x 
P,  CO  x  P,  and  DL  x  P  are  shown  in  Figure  2.  These  help  us  understand 
what  the  numbers  mean.  The  plot  of  the  XP  interaction  shows  exactly 
what  the  investigators  had  purposefully  designed  into  the  experiment. 

In  transfer,  the  three  widely  different  transfer  vehicle  configu¬ 
rations  had  large  differential  effects;  in  training,  where  ^8  unique 
configurations  were  employed,  with  levels  of  individual  factors 
balanced  both  within  and  across  groups,  differences  among  the  three 
groups  were  small,  as  they  should  be.  This  combination  creates  a 
numerical  X  x  P  interaction.  The  large  overall  X  x  P  effect  is  due  to 
the  fact  that  the  averaging  of  the  small  group  differences  during 
training  and  the  large  group  differences  during  transfer  still  resulted 
in  a  large  overall  difference,  but  it  is  meaningless  in  the  light  of 
the  interaction  and  our  knowledge  of  the  experimental  design. 

Our  interest  in  the  large  CO  and  DL  effects  is  overridden  by  the 
existence  of  non-trivial  CO  x  P  and  DL  x  P  interactions.  Both  CO  and 
DL  show  large  effects  during  training  and  relatively  small  effects 
during  transfer.  These  are  the  interaction  effects. 
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Actually  sooe  of  the  smaller  effects  in  this  combined  analysis  are 
of  greater  interest;  the  most  important  of  these  are  the  ones  involving 
an  interaction  with  P.  For  a  factor  not  to  interact  with  P  means  that 
its  effect  is  essentially  the  same  during  the  training  and  transfer 
phases.  When  a  factor  does  interact  with  P,  it  means  that  a  marked 
change  took  place  between  training  and  transfer.  It  is  necessary  to 
determine  whether  this  change  is  just  a  change  in  the  magnitude  of  an 
effect  (as  in  the  cases  of  X,  CO,  and  DL)  or  whether  an  intrinsic 
interaction  exists  in  which  the  effect  of  a  factor  reverses  itself 
between  training  and  transfer.  Only  TM  (Tracking  Mode)  and  PT  (Pre¬ 
diction  Time)  appear  with  P  (Experiment  Phase)  in  an  intrinsic 
interaction. 

In  Figure  2  the  TM  x  P  and  the  TM  x  PT  x  P  interactions  are 
plotted.  In  both  cases,  classic  intrinsic  interaction  can  be  seen. 

The  rank  orders  of  performance  are  inverted  as  a  function  of  whether 
the  measures  were  for  the  training  or  transfer  phase.  The  numbers  tell 
us  that  the  effects,  though  snail,  are  probably  reliable.  In  a  simu- 
lator-to-airplane  experiment,  this  result  could  be  Important  if  valid, 
and  we  would  have  independent  criteria  with  which  to  evaluate  the 
degree  of  importance.  Even  more  pertinent,  such  a  finding  would 
encourage  support  for  the  investigator  to  extend  his  examination  of  the 
phenomenon. 

In  Table  17,  all  sources  of  variance  are  listed  that  show  intrin¬ 
sic  interaction  effects,  i.e.,  a  reversal  in  the  signs  of  their  coef¬ 
ficients  from  training  to  transfer.  With  the  exception  of  the  two  with 
asterisks,  none  of  these  is  statistically  reliable  at  the  level  used 
for  the  stepwise  regression  equation  (namely,  F  =  #.00  to  enter  and 
exit).  These  results  are  shown  here  merely  as  a  matter  of  interest, 
since  whether  they  would  be  important  or  not  depends  on  the  magnitude 
of  the  effect  and  not  on  the  results  of  a  statistical  test  or  the 
proportion  of  variance  accounted  for. 
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TABLE  17.  REVERSALS  IN  PERFORMANCE  LEVELS  AS  A  FUNCTION 
OF  THE  EXPERIMENTAL  PHASE 


Factors  and  factor  interactions  for  which  the  training  coefficient 
is  larger  than  the  transfer  coefficient: 


Training  Coefficient 
was  Negative; 
Transfer  was  Positive 


Training  Coefficient 
was  Positive; 
Transfer  was  Negative 


Training  Trials  Tracking  Mode  x  Control  Gain 

..  Control  Order  x  Display  Lag  .  Prediction  Time  x  Training  Trials 

.  Training  Trials  x  T VC,  Linear 
..  Display  Lag  x  TVC,  Quadratic 
. .  Tracking  Mode*  . . 


Factors  and  factor  interactions  for  which  the  training  coefficient 
is  smaller  than  the  transfer  coefficient: 


Training  Coefficient 
was  Negative; 
Transfer  was  Positive 


Training  Coefficient 
was  Positive; 
Transfer  was  Negative 


Control  Gain  . .  Control  Order  x  TVC,  Linear 

Control  Order  x  Control  Gain  .  Tracking  Mode  x  Prediction  Time*  .. 
Tracking  Mode  x  Training  Trials  . 

Prediction  Time  x  Control  Gain  . 


"Effect  x  Phase  interaction  was  statistically  significant. 

.Number  of  dots  indicates  how  much  greater  in  size  (in  whole  units)  each 
coefficient  was  than  its  standard  error.  Those  to  left  of  term  are  for 
the  training  coefficients  and  to  the  right,  transfer  coefficients. 
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SECTION  VI 

ECONOMICAL  APPROACHES  TO  TRANSFER  OF  TRAINING  RESEARCH 

One  purpose  of  this  study  was  to  discover  relationships  between 
training  and  transfer  performance  that  might  be  generalizable  to  trans¬ 
fer  of  training  experiments  of  the  type  to  be  performed  at  the  VTRS 
laboratory.  Another  purpose  was  to  provide  an  opportunity  to  manipu¬ 
late  the  limited  data  base  to  discover  more  economical  approaches  to 
transfer  of  training  experiments.  Of  these  two  goals,  the  latter  was 
achieved  more  successfully  than  the  former.  A  far  simpler  task  was 
employed  in  the  experiment  than  would  be  expected  in  any  simulator 
likely  to  be  of  interest.  This  probably  resulted  in  a  less  rich  data 
base  than  might  have  been  desired.  As  a  result,  generalizing  results 
regarding  training-transfer  relationships  must  be  done  cautiously  and 
tentatively. 

On  the  other  hand,  the  test  of  a  new  economical  approach  to  trans¬ 
fer  of  training  research  is  exciting  and  can  immediately  promise  better 
and  less  expensive  information  when  applied  to  more  complex  simulation 
problems.  After  35  years  of  simulation  experiments  using  conventional 
transfer  designs,  this  was  the  first  to  examine  more  than  a  few  equip¬ 
ment/training  factors  and  multiple  transfer  vehicle  configurations  in 
the  same  experiment.  The  information  obtained  regarding  both  the 
effectiveness  and  problems  of  this  experimental  plan  is  being  employed 
in  the  planning  of  a  new  transfer  of  training  experiment  at  the  Naval 
Training  Equipment  Center. 

Comparing  Present  Approach  with  Conventional  Approaches 

This  study  illustrates  quite  clearly  the  tremendous  economy  that 
can  be  achieved  with  multifactor  designs  that  ordinarily  obtain 
equivalent  or  better  information  than  conventional  designs.  The  design 
provided  the  data  needed  to  describe  the  relationships  among  six 
equipment/training  factors  and  three  transfer  vehicle  configurations 
using  only  48  data  collection  points.  Eight  additional  data  points 
were  added  to  estimate  the  lack  of  fit,  curvilinearity,  and  error. 
Twenty- four  more  measures  were  made,  making  a  total  of  80  in  all,  to 
obtain  estimates  of  the  average  performance  of  three  control  groups 
with  eight  subjects  each. 

In  the  design,  49  individual  training  configurations  were  examined 
and  an  equation  was  obtained  that  would  approximate  performance  on  any 
combination  of  training  and  transfer  configurations  within  the  experi¬ 
mental  space.  Although  only  one  subject  per  training  configuration  was 
used  in  the  basic  design,  the  mean  of  each  level  of  every  two-level 
factor  in  the  design  was  based  on  24  measurements.  The  means  of  each 
of  the  three  transfer  vehicle  configurations  were  based  on  16  measure¬ 
ments.  How  does  this  compare  with  more  conventional  approaches? 
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One  factor  at  a  time.  Suppose  the  investigator  had  looked  at  the 
effect  of  one  two-level  training  factor  on  one  transfer  configuration 
at  a  time.  The  design  for  each  factor  might  look  like  this: 


Matching 

Training* 

Transfer 

Experimental  Group  1 

Xm 

(1) 

Xt 

Experimental  Group  2 

Xm 

(a) 

Xt 

Control  Group 

Xm 

Xt 

Direct  comparison  of  precision  between  the  one-factor  study  and 
the  multifactor  experiment  i3  not  easy  to  do  without  knowledge  of  the 
error  variance  of  the  two  studies.  Still  some  limiting  calculations 
can  show  how  expensive  the  one-factor-at-a-time  approach  can  be,  in 
both  information  and  resources,  when  compared  with  the  multifactor 
approach.  For  example,  instead  of  requiring  that  the  mean  of  each 
level  of  a  factor  be  based  on  24  observations  as  in  our  multifactor 
experiment,  let  us  require  only  12  observations  (subjects)  per  experi¬ 
mental  group  in  the  one-factor  study.  Thus  each  factor  per  transfer 
configuration  would  require  24  measurements  for  the  two  experimental 
groups. 

With  six  factors  and  three  transfer  configurations,  we  would  need 
2U  x  18  s  43 2  measurements,  plus  24  more  for  eight  subjects  per  the 
three  control  groups,  or  a  total  of  456.  For  this  conservative  number, 
almost  six  times  larger  than  the  current  study,  we  obtain  less  precise 
estimates  of  means  (and  mean  differences)  and  no  estimates  of  any 
interaction  effects.  Also  an  often  overlooked  point  is  that  since  each 
single-factor  study  holds  the  other  factors  at  a  fixed  level,  the 
generalization  of  the  findings  to  a  wide  range  of  conditions  is 
severely  limited. 


•Letters  are  the  conventional  coding  of  each  experimental  condi¬ 
tion,  where  the  presence  of  a  letter  indicates  the  high  level  of  that 
factor  and  the  absence  of  a  letter  indicates  the  low  level  of  the 
factor.  The  (1)  indicates  that  all  factors  are  at  their  lower  level. 
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Two  factors  at  a  time.  Suppose  the  investigator  decides  to  test 
two  factors  at  a  time  to  obtain  both  main  and  two-factor  interaction 
estimates  for  each  of  the  three  transfer  configurations.  Then  the 
design  for  a  single  pair  of  factors  and  one  transfer  configuration 
might  look  like  this: 


Matching 

Training 

Transfer 

Experimental  Group  1 

*m 

(1) 

Xt 

Experimental  Group  2 

*m 

(a) 

Xt 

Experimental  Group  3 

*m 

(b) 

Xt 

Experimental  Group  4 

*m 

(ab) 

Xt 

Control  Group 

*m 

Xt 

To  compare  the  costs  of  this  approach  with  those  of  the  present 
study,  let  us  assign  six  subjects  to  each  of  the  experimental  groups. 
This  again  probably  produces  less  precise  estimates  of  main  and  inter¬ 
action  effects  than  were  obtained  here.  With  six  factors,  there  are  15 
different  pairs  of  two-factor  interactions.  To  collect  the  data  re¬ 
quired  to  estimate  all  two-factor  interactions  for  three  transfer  con¬ 
figurations,  we  would  need  six  subjects  times  four  experimental  groups 
times  three  transfer  configurations,  or  72  for  each  pair  of  factors, 
times  15  pairs  for  a  total  of  1080  data  points.  To  this  we  add  24  more 
subects  with  eight  used  in  each  of  the  three  control  groups.  This  makes 
a  grand  total  of  1104  points. 

With  the  two-factor  approach,  we  obtain  estimates  of  all  main  and 
two-factor  interaction  effects,  as  with  the  present  study,  but  this 
approach  requires  nearly  14  times  the  effort  that  the  present  study 
did.  Furthermore,  the  two-factor  approach  is  probably  less  precise  in 
its  estimates  and  surely  more  biased  because  the  factors  not  included 
in  two- factor  studies  are  held  constant  (see  Simon,  1979,  Section  II). 

The  present  design.  It  can  be  seen  that  as  long  as  the  same 
information  regarding  main  and  two-factor  interaction  effects  is 
required,  the  design  used  will  ordinarily  be  cheaper  and  more  precise 
and  its  results  less  biased  and  more  generalizable  than  comparable  few- 
factors-at-a-time  approaches. 

The  greatest  advantage  of  the  present  design,  however,  is  that  it 
ties  the  information  together  in  a  single  equation  of  specified  preci¬ 
sion;  if  the  Investigator  needs  more  precision,  he  must  collect  more 
data.  Rather  than  looking  at  isolated  segments  of  the  total  space, 
this  design  provides  a  description  over  the  entire  multifactor  space 
within  which  interpolations  can  be  made  with  some  confidence.  This 
means  that  the  data  can  be  generalized  beyond  the  specific  experimental 
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conditions  included  in  the  design,  and  it  is  easier  to  "hang"  new  data 
on  this  original  frame  when  new  factors  are  studied.  This  permits  a 
modular  data  base  to  be  constructed. 

Indications  Regarding  Transfer  of  Training  Experiments 

As  intended,  the  present  study  provides  some  empirical  data  that 
can  be  useful  in  planning  future  transfer  of  training  experiments. 
However,  since  this  is  the  first  study  of  its  kind,  involving  a  rela¬ 
tively  simple  task  and  with  little  learning  and  limited  transfer,  one 
must  consider  bhe  following  observations  as  "indications,"  a  concept  of 
data  analysis  that  Mosteller  and  Tukey  (197?)  bring  to  a  high  level  of 
respectability.* 

In  this  experiment,  however,  several  results  were  observed  that 
are  quite  important  in  the  design  of  transfer  of  training  studies  pro¬ 
vided  they  are  general izable: 

1 .  The  transfer  performance  surface  appears  less  complex 
than  the  training  performance  surface  (see  Tables  4 
and  5 ) . 

2.  The  equipment  factors  in  the  transfer  regression 
equation  (although  specifically  selected  for  their 
potential  importance  to  the  task  under  investigation) 
accounted  for  only  a  small  proportion  or  the  total 
variance. 

If  these  patterns  hold,  then  two  important  practical  conclusions  can  be 
drawn: 

1 .  Fewer  data  will  be  required  to  establish  a  transfer 
performance  surface,  since  a  lower-order  model  is 
indicated.  (A  sequential  strategy,  described  later 

in  this  paper,  ensures  the  most  effective  and  economical 
approach  to  fit  the  appropriate  model  to  the  data. ) 

2.  When  prediction  of  transfer  is  critical,  more  rather 
than  fewer  factors  should  be  investigated  during  the 
screening  phase,  including  task  ani  training  factors 
as  well  as  additional  equipment  variables  not  con¬ 
sidered  in  this  study. 


•Mosteller  and  Tukey  (1977,  Chapter  2)  write:  "The  word  indica¬ 
tion  is  a  vague  concept  intended  to  include,  at  one  extreme,  all  of  the 
classical  descriptive  statistics...  but  also,  at  another  extreme,  to 
include  any  hints  and  suggestions  that  might  prove  informative  to  a 
reasonable  man...  What  indication  is  not  is  inference  or  treatment  of 
uncertainty..."  (pp.  25-27). 
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Economy  as  a  Function  of  Purpose 

The  purpose  for  which  a  study  is  performed  can  markedly  affect  the 
amount  of  data  collection  required,  independent  of  the  number  of  fac¬ 
tors  being  investigated.  If  the  study  is  done  primarily  to  identify 
the  important  factors,  fewer  data  will  be  required  than  if  its  purpose 
is  to  3ay  with  reasonable  confidence  that  slight  differences  are  in 
fact  mt  real  differences  between  levels.  In  the  latter  case,  the 
power  of  the  test  requires  many  more  observations  to  accept  the  null 
hypothesis  with  reasonable  confidence. 

Similarly,  fewer  data  are  ordinarily  required  in  experiments 
intended  to  identify  critical  factors  rather  than  to  write  an  equation 
representing  a  response  surface.  Generally,  the  former  can  be  satis¬ 
fied  with  a  modified  fractional  factorial  design;  the  latter  requires 
that  the  data  collection  process  continue  until  the  model  being  created 
fit3  the  experimental  data.  The  present  design  represents  a  compromise 
between  these  two  purposes.  By  adding  the  centerpoint3,  lack  of  fit 
and  curvilinearity  can  be  evaluated,  yet  each  factor  cannot  be  identi¬ 
fied  individually. 

Of  particular  interest  to  transfer  of  training  experiments,  where 
economy  is  a  still  more  serious  consideration,  is  the  distinction 
between  an  experiment  intended  only  to  identify  which  factor  affects 
transfer  of  training  the  most  and  one  that  must  provide  a  measure  of 
transfer.  In  Table  13,  the  coefficients  of  each  factor  (with  the 
exception  of  Transfer  Vehicle  Configuration)  are  identical  for  the 
transfer  performance  scores  and  the  transfer  effect  scores  in  which  the 
mean  performance  of  each  control  group  has  been  subtracted  from  the 
transfer  performance  score  of  the  corresponding  transfer  group.  Thus 
the  mean  difference  between  levels  for  the  individual  equipment  vari¬ 
ables  is  the  same  whether  or  not  a  control  group  is  used. 

From  the  standpoint  of  design,  this  means  that  we  can  predict 
which  equipment  combination  will  yield  the  highest  performance  on  the 
transfer  configuration  without  collecting  independent  control  data.  In 
the  present  study,  this  would  have  represented  a  savings  of  30*. 

Roscoe  0980,  Chapters  16  and  17)  has  discussed  the  various  ways  of 
measuring  the  transfer  and  cost  effectiveness  of  simulator  training. 
However,  in  transfer  of  training  studies  for  simulator  design  purposes, 
the  raw  performance  scores  are  sufficient  during  the  early  "screening" 
phase,  in  which  the  purpose  is  simply  to  identify  the  best  configura¬ 
tion.  It  is  in  the  evaluation  phase  later  in  a  program  that  experi¬ 
ments  should  be  performed  with  control  groups. 
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Predicting  Transfer  from  Training  Performance  Data 

If  it  were  possible  to  eliminate  the  transfer  phase  of  a  training 
study  considerable  savings  would  be  elicited.  This  could  be  done  if  it 
were  found  that  a  significant  relationship  exists  between  training  per¬ 
formance  scores  and  transfer.  The  observed  correlation  of  .37  between 
the  56  training  performance  scores  and  the  transfer  effect  scores,  with 
54  degrees  of  freedom,  would  be  expected  to  occur  only  about  five  times 
in  1000  by  chance.  Nevertheless,  an  r  of  .37  accounts  for  less  than  14 
perceht  of  the  variability  exhibited  among  the  transfer  effect  scores 
and  therefore  has  relatively  little  practical  prediction  value. 

Applying  Principles  of  Transfer  to  Achieve  Economy 

Some  psychologists  believe  they  can  predict  the  transfer  potential 
of  simulator  configurations  on  the  basis  of  principles  that  have  been 
proposed  at  one  time  or  another  regarding  the  necessary  relationships 
between  training  and  transfer  (see  Roscoe,  1980,  Chapters  15-22).  If 
this  were  in  fact  possible  this  could  eliminate  or  at  least  reduce  the 
data  collection  effort  required  to  make  decisions  regarding  the  design 
of  the  training  simulator.  However,  at  present,  these  principles  are 
at  best  imprecise  generalizations,  largely  a  product  of  unstructured 
empirical  observation  and  tenuous  analogy  with  findings  from  abstract 
laboratory  experiments  on  verbal  and  motor  learning.  Although  the 
present  study  was  not  designed  to  isolate  individual  principles  of 
transfer,  some  information  regarding  so-called  transfer  principles  was 
obtained. 

The  training  performance  principle.  It  has  been  suggested  that 
the  size  of  a  transfer  of  training  experiment  might  be  reduced  by 
eliminating  those  factors  during  transfer  that  fail  to  influence  per¬ 
formance  critically  during  the  training  phase.  This  principle  may  be 
stated  as  follows:  Factors  that  fail  to  influence  performance  during 
training  will  not  critically  influence  performance  during  transfer. 
Since  in  holistic  experiments  involving  ten  or  more  factors  it  is  not 
uncommon  to  find  many  noncritical  effects,  were  this  theory  a  valid  and 
sufficient  one,  considerable  savings  could  be  elicited. 

But  there  are  reasons  to  suspect  the  validity  of  this  principle. 
Simon  (1971)  has  discussed  how  task  difficulty  is  a  hidden  variable 
that  can  confound  the  magnitude  of  the  effects  of  the  manipulated 
variables.  By  way  of  illustration  for  certain  situations,  if  a  task  is 
too  easy,  no  differences  may  be  observed  in  the  performance  obtained  at 
two  levels  of  a  factor.  As  the  task  becomes  more  difficult,  differ¬ 
ences  in  performance  at  the  two  levels  will  begin  to  appear.  When  the 
task  becomes  too  difficult,  performances  at  the  two  levels  will  become 
the  same  again. 
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In  transfer,  quite  frequently  the  overall  operational  task  will  be 
more  complex  and  difficult  than  the  simulated  task  due  to  adverse 
environmental  factors  not  represented  in  the  simulation.  Carryover 
effects,  another  name  for  transfer,  can  be  confounded  with  task  diffi¬ 
culty  level  in  the  same  way.  If  this  is  the  case,  we  cannot  say  with¬ 
out  a  great  deal  of  additional  knowledge  whether  effects  will  or  won't 
be  larger  or  smaller  between  training  and  transfer. 

If  one  examines  the  experimental  results  (Tables  12  and  13),  it 
can  be  observed  that  between  the  training  and  transfer  phases  some 
large  effects  become  small  and  some  small  effects  become  large.  It 
has  already  been  noted  that  the  total  transfer  surface,  except  as 
affected  by  the  transfer  vehicle  configurations,  was  relatively  flat. 
Still,  an  inspection  of  the  data  can  be  enlightening.  In  Table  18  are 
listed  all  statistically  significant  interaction  coefficients  for 
transfer  performance  from  Table  13  and  the  corresponding  training  per¬ 
formance  coefficients  from  Table  12. 


TABLE  18.  INTERACTION  TERMS  THAT  WERE  STATISTICALLY  SIGNIFICANT 
(£  <  .05)  IN  TRANSFER  PHASE* 


Coefficients  and  Chance  Probability 


Training 

£ 

Transfer 

£ 

Ratio 

DL  x  TT 

6.2 

.644 

35.7 

.040 

5.8 

TM  x  PT 

19.2 

.155 

-39.1 

.026 

2.0 

TM  x  XL 

-29.9 

.065 

-58.2 

.006 

1.9 

TM  x  XQ 

5.9 

.509 

25.5 

.029 

4.3 

CG  x  TT 

-9.6 

.477 

-46.7 

.009 

4.9 

CG  x  XQ 

10.4 

.250 

24.7 

.034 

2.4 

•From  Tables  12  and  13 


There  are  six  interaction  effects  that  were  not  larger  than  might 
have  been  expected  by  chance  during  training  but  were  during  transfer. 
The  coefficients  in  these  cases  increased  from  two  to  six  times  in 
magnitude.  While  this  study,  involving  a  relatively  simple  perceptual- 
motor  task,  may  not  be  representative  of  the  carrier  landing  task  or  a 
simulator's  visual  "fidelity"  factors,  it  does  provide  an  empirical 
indication  of  the  danger  of  accepting  the  theory  that  what  isn't 
important  during  training  will  not  be  important  for  transfer. 

Lincoln  (1978)  performed  a  transfer  of  training  experiment  in¬ 
volving.  four  longitudinal  compensatory  tracking  tasks  that  differed 
with  regard  to  system  dynamics.  Two  factors  at  two  levels  each  were 
investigated:  short  period  natural  frequency  and  damping  character¬ 
istics.  In  discussing  the  results  of  the  study,  he  wrote: 


"The  most  important  result  of  this  study  concerns 
the  differential  effects  of  the  two  major  dynamic 
variables.  One  of  them,  damping,  greatly  influenced 
task  difficulty  but  was  of  little  importance  in 
determining  the  effects  of  transfer  of  training. 

The  second  variable,  natural  frequency  of  the  system 
affected  performance  in  exactly  the  reverse  manner. 
Its  influence  was  relatively  unimportant  with  re¬ 
gard  to  task  difficulty,  but  it  appeared  to  be  of 
primary  importance  in  determining  the  amount  of 
transfer  of  training  that  occurred"  (p.  88). 


The  la3t  statement  is  another  empirical  example  wherein  the  theory 
proposed  at  the  beginning  of  this  section  was  contradicted.* 


•In  support  of  the  holistic  approach  to  equipment  design  research, 
the  following  quotation  was  taken  from  the  discussion  in  Lincoln’s 
paper:  ’  "Unfortunately  for  the  designer,  as  Muckier,  Obermayer,  Hanlon, 
and  Serio  (1961)  have  shown,  the  complex  nature  of  transfer  effects  and 
the  need  to  recognize  the  possiblity  that  other  variables  may  interact 
with  frequency  and  damping,  makes  broad  generalizations  dangerous.  In 
their  second  report,  these  investigators  found  that  control  gain  set¬ 
tings  could  drastically  alter  the  patterns  of  transfer  that  were  ob¬ 
served.  It  appears,  therefore,  that  designers  of  manual  control  systems 
are  presently  faced  with  problems  of  considerable  complexity  with  only 
limited  empirical  information  on  which  to  base  their  design  decisions. 

A  continuing  effort  to  untangle  the  interrelated  effects  of  training  and 
transfer  would  seem  to  be  appropriate"  (p.  89). 
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The  similarity  principle.  One  of  the  moat  frequently  stated 
training  and  transfer  generalizations  is  that  associated  with  the 
similarity  between  training  and  transfer  conditions,  commonly  referred 
to  as  "fidelity"  when  applied  to  simulators.  This  principle  asserts: 

Transfer  of  training  from  simulator  to  air¬ 
craft  is  a  positive  function  of  the  degree 
to  which  the  simulator  faithfully  reflects 
the  characteristics  of  the  aircraft. 

The  effect  of  this  principle  has  been  the  design  and  development  of 
simulators  that  have  a  high  degree  of  physical  similarity  to  the  real 
airplane,  a  costly  decision  of  unknown  payoff.  Still,  it  is  generally 
recognized  (but  sometimes  forgotten)  that  task  similarity  does  not 
necessarily  depend  on  a  faithful  physical  representation  of  reality. 
What  it  does  depend  on  necessarily  is  a  faithful  representation  of  the 
responses  that  must  be  learned,  and  the  conditions  needed  to  elicit 
those  responses. 

The  chief  problem  with  similarity  (or  fidelity)  in  a  complex 
system  in  a  real-world  environment  is  that  it  is  not  always  easy  to 
define  or  measure.  In  fact,  as  Simon  (1979,  Section  VI)  has  noted,  it 
is  a  multivariate  concept  that  differs  in  form  and  meaning  for  differ¬ 
ent  physical  components  of  a  complex  simulation.  Then  too,  only  cer¬ 
tain  components  are  critical  in  simulating  particular  tasks,  and  as  yet 
there  are  no  adequate  principles  for  deciding  which  are  and  which 
aren't  critical.  The  concept  of  similarity  is  further  complicated  by 
the  interaction  between  stimulus  similarity  and  response  similarity  and 
their  differential  effects  on  positive  and  negative  transfer. 

To  make  matters  worse,  these  are  not  the  only  principles  that  have 
an  effect  on  training  and  transfer,  and  others,  such  as  adaptive  feed¬ 
back,  may  override  the  effects  of  similarity  in  any  situation.  Until 
these  principles  have  been  adequately  dimensionalized  and  empirically 
evaluated  together  and  in  context,  they  will  continue  to  offer  only 
superficial  aid  in  the  design  of  complex  simulators  that  are  optimized 
both  for  pilot  training  and  for  cost  of  ownership,  maintenance,  and 
operation. 

From  the  results  of  the  present  experiment,  two  dimensions  of  task 
similarity  could  be  examined,  factor  fidelity,  as  represented  by  the 
distance  between  levels  of  the  various  experimental  factors,  and  rela¬ 
tive  difficulty,  as  represented  by  the  different  training  and  transfer 
vehicle  configurations.  Stated  in  its  negative  form,  the  fidelity 
principle  asserv?  that:  The  farther  apart  the  training  and  transfer 
levels  of  a  particular  factor  are,  the  lower  the  transfer  effect.  The 
relative  difficulty  principle  asserts  that:  More  positive  transfer 
will  be  elicited  when  task  difficulty  shifts  from  hard  to  easy  than 
from  easy  to  hard. 
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The  transformations  used  to  change  coded  experimental  coefficients 
of  each  equipment/training  factor  to  "fidelity"  and  "difficulty"  coef¬ 
ficients  are  shown  in  Table  19.  With  these  "similarity"  coefficients 
the  experimental  design  is  no  longer  orthogonal.  Still  a  stepwise 
regression  analysis  could  be  and  was  performed  to  see  which  of  the  six 
fidelity  (factor-distance)  and  six  difficulty  (factor-direction) 
variables  most  influenced  transfer  performance.  With  an  F  of  U.00  to 
enter  and  exit  the-  equation,  only  three  of  the  12  terms  appeared  when 
transfer  performance  scores  were  analyzed  and  only  two  terms  when 
transfer  effect  scores  were  the  criterion. 


TABLE  19.  TRANSFORMATIONS  USED  TO  CHANGE  CODED  EXPERIMENTAL 
COEFFICIENTS  TO  SIMILARITY  COEFFICIENTS  FOR  EACH 
FACTOR  AND  CONDITION 


Coded 

Experimental  Coefficients 


Coded  Similarity  Coefficients 


Fidelity 

(Distance) 


Difficulty 

(Direction) 


Training 

Transfer 

Code 

Code 

Actual 

Code 

Actual 

Code 

-1  (Hard)* 

-1  (Hard) 

0 

-1 

No  Change 

0 

-1 

0  (Central) 

1 

0 

Hard  to  Easy 

-1 

-1 

+1  (Easy) 

2 

+  1 

Hard  to  Easy 

-1 

♦1  (Easy) 

-1 

2 

+  1 

Easy  to  Hard 

+  1 

>1 

0 

1 

0 

Easy  to  Hard 

+  1 

>1 

♦  1 

0 

-1 

No  Change 

0 

0  (Central) 

-1 

1 

0 

Easy  to  Hard 

+  1 

0 

0 

0 

-1 

No  Change 

0 

0 

+  1 

1 

0 

Hard  to  Easy 

-1 

•These  are  the  preexperimental  assumptions  regarding  the  relative 
difficulty  of  performing  at  this  level. 
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Table  20  lists  the  terms  that  appeared,  their  coefficients  in  the 
equations,  the  incremental  proportion  of  variance  each  contributed,  the 
standard  regression  coefficients  of  each,  and  such  summary  statistics 
as  the  F-value,  the  standard  error  of  estimate,  the  standard  error  of 
coefficients,  and  the  adjusted  R2. 


TABLE  20.  SUMMARY  OF  STEPWISE  REGRESSION  ANALYSES  OF 
INTERVENING  FIDELITY  AND  DIFFICULTY  FACTORS 
WITH  AN  F  OF  4.00  TO  ENTER  AND  EXIT 


Transfer  Performance 

Scores 

Proportion 

Standard 

Source 

Coefficients 

of  Variance 

Coefficients 

Intercept 

1688.0 

Direction-TM 

♦157.0 

.23 

.34 

Direction-CG 

♦157.0 

.14 

.34 

Direction-CO 

♦106.0 

.05 

.23 

Adjusted  R2  =  .38 

F  =  12.44  (3/52),  £  <  .001 
Standard  Error  of  Estimate  =  278 
Standard  Error  of  Coefficients  =  57.7 

Transfer  Effects 


Intercept 

-11.3 

Distance-TM 

+61.9 

.11 

.33 

Direction-TM 

♦54.0 

.07 

.26 

Adjusted  R2  =  .15 

F  =  5.73  (2/53) 

,  £  <  .01 

Standard  Error 

of  Estimate  =  145 

Standard  Error 

of  Coefficients  =  25.5 
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When  training  performance  scores  were  introduced  as  an  independent 
variable  along  with  the  12  similarity  variables  and  a  stepwise  regres¬ 
sion  analysis  was  performed  for  transfer  performance  and  transfer 
effect  scores,  the  results  shown  in  Table  21  were  obtained: 


TABLE  21.  SUMMARY  OF  STEPWISE  REGRESSION  ANALYSES  OF 

INTERVENING  FIDELITY  AND  DIFFICULTY  FACTORS  WITH 
TRAINING  PERFORMANCE  SCORES  AS  AN  INDEPENDENT 
VARIABLE  AND  AN  F  OF  4.00  TO  ENTER  AND  EXIT 


Transfer  Performance  Scores 


Source 

Coefficients 

Proportion  Standard 

of  Variance  Coefficients 

Intercept 

874.9 

Training  Performance 

■►0.475 

.09  .30 

F  =  5.16  (1/54), 

£  <  .0 

Adjusted  R2  =  .07 

Standard  Error  of 

Estimate  =  34 1 

Standard  Error  of 

Coefficients  =  .209 

Transfer  Effects 

Intercept 

-429.1 

Training  Performance 

♦0.243 

.14  .34 

Direction-TM 

♦57.9 

.08  .27 

Distance-TM 

♦51.1 

.07  .28 

Adjusted  F?2  =  .25 

F  =  7.06  (3/52),  £  <  .0 
Standard  Error  of  Estimate  =  136.4 

Standard  Error  of  Training  Performance  Coefficient  =  0.085 
Standard  Error  of  Coefficients  =  23.0 


Several  results  important  for  simulator  design  ara  indicated  by  the 
above  statistics: 

1.  The  positive  coefficients  of  the  similarity  variables  indicate 
that  the  results  agreed  with  the  principles  stated  earlier. 

For  those  factors  that  had  an  effect,  when  the  factor  level 
between  training  and  transfer  changed  from  hard  to  easy  there 
was  a  greater  reduction  in  RMS  error  than  when  it  changed  from 
easy  to  hard.  Also,  the  shorter  the  distance  between  the  two 
levels,  the  greater  the  reduction  in  RMS  error. 
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2.  In  this  study,  the  direction  of  change  in  difficulty  had 
stronger  effects  than  the  distance  of  change  in  factor  levels. 

3.  The  adjusted  R2  values  showed  that  the  similarity  variables  did 
not  improve  our  prediction  of  transfer  scores  over  that  based 
on  training  performance  scores.  In  fact,  there  was  an  interac¬ 
tive  effect  depending  on  whether  training  performance  or  simi¬ 
larity  variables  were  used  to  predict  transfer  performance  or 
transfer  effect  scores.  None  of  the  combinations  did  as  well 
as  the  original  equipment/training  variables. 

No  generalizations  regarding  the  usefulness  of  similarity  vari¬ 
ables  as  intervening  factors  can  be  drawn  here.  Before  one  can  discard 
these  approaches,  however,  the  following  limitations  of  this  analysis 
must  be  considered:  (a)  It  was  an  adjunct  effort,  an  afterthought,  and 
the  study  was  not  designed  to  obtain  this  kind  of  information,  (b)  The 
performance  and  similarity  variables  in  this  study  were  intercorre- 
lated,  which  means  that  these  solutions  are  not  unique,  (c)  The  meas¬ 
ures  of  similarity  used  here  were  unsophisticated,  (d)  The  similarity 
variables  are  only  two  of  the  possible  classes  that  could  have  been 
included,  (e)  No  interaction  effects  among  the  similarity  variables 
were  included  in  these  analyses. 

The  primary  importance  of  this  exercise  lies  in  the  fact  that  it 
illustrates  what  might  be  done  were  a  comprehensive  set  of  intervening 
factors  developed  and  a  primary  investigation  performed.  The  informa¬ 
tion  obtained  from  such  a  study  would  help  resolve  the  question  of  the 
contribution  of  intervening  factors  to  predictions  between  simulators 
and  test  vehicles. 

Fidelity  and  Quasi-Transfer  Experiments.  The  costs  of  conducting 
simulator-to-air  transfer  studies  have  severely  limited  that  type  of 
research.  Then  too,  certain  difficult-to-accept  risks  exist  when  a 
pilot  must  be  tested  in  the  air  after  having  been  trained  on  a  less 
than  optimum  simulator  training  configuration.  For  these  reasons, 
quasi- transfer  experiments,  in  which  both  training  and  transfer  occur 
in  simulators,  can  fill  an  important  gap.  The  problem  with  quasi¬ 
transfer  studies  is  that  their  results  can  be  questioned  on  the  grounds 
that  the  observed  transfer  was  to  a  simulator,  not  to  an  airplane. 

Unless  the  investigator  can  show  that  performance  in  the  transfer 
configuration  of  the  simulator  does  not  differ  in  any  critical  way  from 
performance  in  the  airplane  there  always  will  be  room  for  doubt  regarding 
the  information  a  quasi-transfer  experiment  provides.  For  this  reason, 
if  transfer  of  training  research  is  anticipated  and  if  economy  is  an 
issue,  then  some  secondary  effort  must  be  devoted  to  establishing  the 
relationship  between  performance  in  the  most  physically  faithful  configu¬ 
ration  of  the  simulator  and  performance  in  the  aircraft.  It  is  also 
important  to  develop  multidimensional  similarity  principles  that  will 
permit  the  transfer  relationships  between  simulator  and  aircraft  to  be 
defined  quantitatively. 
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In  the  carrier  landing  performance  study  carried  out  earlier  at 
the  VTRS  laboratory  (Westra,  et  al . ,  1981),  128  experimental  conditions 
were  used  to  evaluate  the  effects  of  ten  equipment  and  environmental 
factors  on  carrier  landing.  The  likelihood  of  running  a  transfer  of 
training  experiment  of  that  size  is  meager,  both  because  of  the 
extended  costs  of  training  and  the  difficulty  of  finding  the  large 
number  of  subjects  that  would  be  needed. 


Ten  factors  can  still  be  investigated,  however,  using  only  32 
experimental  conditions.  One  limitation  of  such  a -design  is  that  32 
conditions  are  close  to  the  minimum  that  would  provide  an  acceptable 
level  of  precision.  Each  mean  is  based  on  16  observations,  which  is  a 
reasonable  number.  However,  with  highly  variable  data,  our  confidence 
limits  will  be  quite  wide.  Still  the  design  can  pick  out  those  factors 
that  have  an  important  effect  on  performance  but  may  miss  the  marginal 
ones.  Another  limitation  of  a  design  of  that  size  is  that,  while  it 
isolates  all  main  effects  from  each  other  and  from  all  two-factor 
interactions,  i.t  does  not  isolate  all  of  the  two- factor  interaction 
effects  from  one  another.  The  two-factor  interactions  are  aliased  in 
strings. 


If  32  experimental  conditions  are  too  costly,  there  are  3till  more 
economical  designs  that  might  be  employed.  It  is  important,  however, 
to  emphasize  that  one  uses  these  designs  to  obtain  the  best  empirical 
information  possible  under  extremely  limited  conditions,  and  that  the 
best  may  or  may  not  be  suitable  for  certain  purposes.  The  decision  to 
use  such  designs  is  not  an  experimental  question.  It  is  one  that 
management,  informed  of  the  tradeoffs  involved,  must  make. 

For  example,  there  are  main-effect  designs  by  Plackett  and  Burman 
( 19*16)  (see  Simon,  1973)  that  increase  in  steps  of  four  rather  than 
powers  of  two.  Thus  Plackett-Burman  designs  are  based  on  4,  8,  12,  16, 
20,  24,  28,  32  conditions  rather  than  4,  8,  16,  32  conditions  of  the 
usual  fractional  factorial  plans.  With  ten  factors,  for  example,  one 
can  measure  all  main  effects  using  only  12  experimental  conditions. 
However,  these  effects  will  be  partially  confounded  with  two-factor 
interactions  (r  *  +  0.333)  and  with  three-factor  interactions  (r  = 

♦  0.354) .•  Furthermore,  each  interaction  will  be  found  in  several 
strings,  each  partially  confounded  with  a  number  of  other  effects. 

In  the  standard  2*"?  designs,  confounding— if  it  occurs — is  total 
(r  s  ♦  1.0)  and  unique  to  a  particular  set  of  other  effects. 


•Dr.  David  Weinman,  Hollins  College,  Virginia  provided  the  infor¬ 
mation  regarding  the  intercorrelations  for  the  Plackett-Burman  designs. 


With  these  Plackett-Burman  designs,  by  adding  12  more  experimental 
conditions  —  the  "foldover"  (Simon,  1973)  of  the  original  plan  —  the 
main  effects  can  now  not  only  be  isolated  from  one  another  but  also 
from  all  two-factor  interactions.  The  two-factor  interactions  remain 
partially  correlated  with  other  two-factor  interactions  in  several 
different  strings,  but  the  cost  has  been  reduced  from  32  to  24  condi¬ 
tions.  Only  the  main  effects  remain  correlated  with  the  three-factor 
interactions  (r  =  ♦  0.333). 

Thus  for  certain  numbers  of  factors,  Plackett-Burman  designs 
enable  smaller  studies  to  be  done  than  the  2k_p  designp.  In  the  above 
example,  there  was  a  25%  savings  in  data  collection.  The  precision 
with  which  the  means  can  be  measured  dropped  13t  with  the  smaller 
design.  The  higher-order  effects  become  more  difficult  to  untangle 
than  when  fully  aliased.  On  the  other  hand,  being  partially  corre¬ 
lated,  no  single  large  interaction  effect  is  likely  to  bias  the  main 
effect  estimates  seriously.  With  a  correlation  of  0.333,  the  overlap 
is  approximately  lit.  These  designs  might  be  considered  when  it  is 
essential  that  the  3tudy  be  kept  small,  when  less  precision  can  be 
accepted  regarding  the  main  effects,  and  when  there  is  little  likeli¬ 
hood  that  there  will  be  a  subsequent  need  to  isolate  interaction 
effects. 

Using  a  Sequential  Strategy 

In  the  present  study,  a  fixed  design  was  employed.  That  is,  the 
size  and  form  of  the  design,  its  resolution  and  other  characteristics 
were  selected  before  the  experiment  began.  This  is  not  the  way  to  per¬ 
form  holistic  experiments  economically.  Instead,  only  the  data  needed 
to  fit  the  lowest-order  surface  should  be  collected  initially,  and  the 
model  tested  to  see  how  well  it  fits  the  empirical  data.  Then  if  the 
fit  is  poor,  more  data  would  be  collected  to  fit  the  next  higher-order 
surface.  This  process  would  continue  until  the  fit  is  adequate.  In 
the  present  study,  the  fixed  design  was  used  because  of  concern  that 
the  time  and  dollar  limitations  might  prematurely  terminate  a  sequen¬ 
tial  effort.  Consequently,  a  plan  was  selected  that  would  guarantee  at 
least  a  Resolution  V  design. 

What  the  present  study  suggests  regarding  the  application  of  a 
sequential  approach  to  transfer  of  training  is  that  we  should  build 
increasingly  complex  models  on  the  basis  of  transfer  scores  rather  than 
training  scores.  This  is  indicated,  at  least  in  the  current  study, 
because  the  transfer  surface  is  less  complex  than  the  training  surface 
and  therefore  requires  fewer  data  points.  Then  too,  transfer  scores 
are  the  values  of  ultimate  interest.  Only  additional  effort  will 
determine  to  what  extent  the  results  of  the  present  experiment  can  be 
generalized,  but  for  the  present,  the  approach  employed  is  appropriate. 
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SECTION  VII 

CONCLUSIONS  AND  DISCUSSION 

This  study  demonstrated  an  efficient  and  economical  approach  for 
collecting  multifactor,  multicriterion  transfer  of  training  data.  The 
approach  is  particularly  useful  in  the  early  stages  of  a  simulator 
design  program,  such  as  VTRS,  when  many  alternatives  should  be  con¬ 
sidered  and  the  individual  contributions  of  component  design  variables 
should  be  evaluated  separately  from  overall  simulator  effectiveness. 

The  unique  nature  of  this  study  should  be  noted.  It  was  the  first 
transfer  of  training  experiment: 

1.  To  examine  as  many  as  six  equipment/training  factors  in  a 
single  study. 

2.  To  examine  a  broad  spectrum  of  training  configurations  at  one 
time.  Forty-nine  configurations  were  examined  here. 

3.  To  train  only  a  single  subject  on  each  of  the  48  main  training 
vehicle  configurations,  reducing  the  number  required  without  sacrific¬ 
ing  precision. 

4.  To  use  more  than  one  transfer  vehicle  configuration  in  a 
single  experiment.  Three  were  used  here. 

5.  To  provide  data  in  equation  form  that  would  answer  specific 
questions  regarding  transfer  from  various  training  simulator  configu¬ 
rations  not  directly  investigated. 

The  cost  of  this  study  in  data-collection  time  was  approximately 
one-eighth  of  what  it  might  have  cost  to  study  the  effects  of  the  six 
factors  and  the  three  transfer  configurations  in  a  series  of  two-factor 
studies  to  obtain  all  main  and  two-factor  interaction  effects.  The 
precision  of  this  study  was  higher  and  potential  bias  less. 

Multifactor  data  collection  plans  of  this  type  are  most  effective 
in  the  early  phases  of  a  simulator  design  program  before  specific  con¬ 
figurations  have  been  selected.  Conventional  transfer  of  training 
research  designs  are  better  fitted  for  use  at  the  end  of  a  design  pro¬ 
gram  when  a  few  configurations  have  been  selected  and  the  objective  is 
to  quantify  the  transfer  effectiveness  of  each. 

The  application  of  this  data  collection  plan  provides  a  number  of 
practical  features  seldom  obtained  from  conventional  transfer  of  train¬ 
ing  experiments  or  rational  analyses  by  design  engineers  and  psycholo¬ 
gists.  For  example,  in  the  early  stages  of  a  simulator  design  program, 
the  actual  features  of  the  transfer  criterion  vehicle  and  consequently 
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the  simulator  requirements  may  not  be  firm.  This  data  collection  plan 
provides  transfer  data  across  a  broad  spectrum  of  conditions  (training 
and  test)  so  that  when  the  airplane  features  are  eventually  selected, 
relevant  transfer  data  will  be  available. 

Isolating  the  effects  of  potentially  critical  variables  in  the 
simulator  provides  better  transfer  data  with  which  to  make  engineering 
decisions  than  do  gross  measurements  of  total  simulator  effectiveness. 
It  enables  the  designer  to  identify  negative  contributions  of  specific 
components  that  might  otherwise  be  hidden  by  positive  overall  results. 
The  data  collection  plan,  by  providing  multifactor  data  in  equation 
form,  not  only  allows  estimates  to  be  made  of  the  effectiveness  of 
simulator  configurations  not  investigated  in  the  study,  but  also 
provides  an  overview  that  enables  tradeoffs  to  be  made  more  precisely 
between  system  performance  and  system  costs. 

The  following  results  in  this  study  have  direct  applications  for 
future  transfer-of-training  efforts  provided  subsequent  investigations 
demonstrate  their  generality: 

1 .  The  transfer  performance  surface  is  approximated  by  a  lower- 
order  model  than  that  required  for  the  training  performance  surface  and 
should  be  used  as  the  criterion  for  collecting  more  data  if  a  sequen¬ 
tial  data  collection  plan  is  employed  to  obtain  maximum  economy. 

2.  The  correlation  between  training  and  transfer  performance  for 
different  design  configurations  was  positive  but  too  low  for  practical 
predictive  purposes. 

3.  Effects  that  are  strong  in  training  may  not  be  strong  in 
transfer  and  vice  versa. 

4.  Intervening  similarity  factors  combined  with  performance 
scores  may  Increase  predictability,  although  insufficient  experimental 
data  exist  to  isolate  and  evaluate  intervening  predictive  factors  at 
this  time. 

5.  Where  time,  pilot  availability,  and  dollars  are  extremely 
limited,  marginal  experimental  plans  can  provide  empirical  data 
regarding  transfer  effectiveness  of  simulator  design  variables,  but 
with  increased  risk  of  error. 

6.  Additional  economy  may  be  achieved  by  eliminating  control 
groups  in  the  early  phases  of  a  simulation  design  program  when  the 
purpose  is  to  select  those  combinations  of  factor  levels  that  produce 
the  highest  transfer  possible  rather  than  measure  a  particular 
configuration's  effectiveness. 
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APPENDIX  A 

TRAINING  VEHir  CONFIGORATIONS  IN  THE  FOUR  EXPERIMENTAL  GROUPS  EXPRESSED 
IN  CODED  Vi^..,iS  CORRESPONDING  TO  REAL-WORLD  VALUES  SHOWN  IN  TABLE  1 

TABLE  A1 

UNIQUE  TRAINING  VEHICLE  CONFIGURATIONS  (COMBINATIONS  OF 
♦  AND  -  FACTOR  LEVELS)  FOR  THE  16  INDIVIDUAL  PARTICIPANTS  IN 
GROUP  A  WHO  TRANSFERRED  TO  THE  HARD  TRANSFER  VEHICLE  CONFIGURATION 


GROUP  A 

Training  Vehicle  Configuration 

Training  Trials 

Participant 

CO  DL  TM  PT  CG 

TT 

01 

- 

- 

* 

- 

- 

- 

02 

♦ 

4 

4 

4 

4 

4 

03 

4 

- 

- 

• 

4 

4 

04 

- 

4 

4 

4 

- 

- 

05 

- 

4 

- 

- 

4 

4 

06 

4 

- 

4 

4 

- 

- 

07 

4 

♦ 

- 

- 

- 

- 

08 

- 

- 

4 

4 

4 

4 

09 

- 

- 

4 

- 

4 

- 

10 

4 

4 

- 

4 

- 

4 

11 

♦ 

- 

4 

- 

- 

4 

12 

- 

4 

- 

4 

4 

- 

13 

- 

4* 

4 

- 

- 

4 

14 

♦ 

- 

- 

4 

4 

- 

15 

4 

4 

4 

- 

4 

- 

16 

4 

4 
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TABLE  A2 

UNIQUE  TRAINING  VEHICLE  CONFIGURATIONS  (COMBINATIONS  OF 


4> 

AND  - 

FACTOR  LEVELS) 

FOR 

THE  16  INDIVIDUAL  PARTICIPANTS  IN 

GROUP 

B  WHO 

TRANSFERRED  TO 

THE 

CENTRAL  TRANSFER  VEHICLE 

CONFIGURATION 

GROUP 

B 

Training  Vehicle  Configuration 

Training  Trials 

Participant 

CO  DL 

TM  PT  CG 

TT 

17 

• 

• 

-  ♦  - 

18 

♦ 

4- 

4-  -  ♦ 

4> 

19 

♦ 

- 

-4-4* 

4* 

20 

♦ 

4*  -  - 

- 

21 

- 

♦ 

-4-4- 

22 

♦ 

- 

4-  - 

- 

23 

♦ 

-  4-  - 

- 

24 

- 

- 

+  -  4- 

4* 

25 

- 

- 

4-  4-  4* 

• 

26 

4* 

4- 

- 

4* 

27 

♦ 

- 

4*  4- 

4* 

28 

- 

4* 

-  -  4- 

- 

29 

- 

♦ 

4-  4-  - 

4» 

30 

4> 

- 

-  -  4- 

- 

31 

4* 

4* 

4>  4-  ♦ 

- 

32 

• 

_  _  _ 

4> 
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TABLE  A3 


UNIQUE  TRAINING  VEHICLE  CONFIGURATIONS  (COMBINATIONS  OF 
+  AND  -  FACTOR  LEVELS)  FOR  THE  16  INDIVIDUAL  PARTICIPANTS  IN 
GROUP  C  WHO  TRANSFERRED  TO  THE  EASY  TRANSFER  VEHICLE  CONFIGURATION 


GROUP  C 


Training  Vehicle  Configuration  Training  Trials 


Participant  CO  DL  TM  PT  CG 


TT 


33 

4 

- 

- 

- 

- 

4 

31* 

- 

4 

4 

4 

4 

- 

35 

- 

- 

- 

• 

4 

- 

36 

4 

4 

4 

4 

- 

4 

37 

4 

4 

- 

- 

4 

- 

38 

- 

- 

4 

4 

- 

4 

39 

- 

4 

- 

- 

- 

4 

40 

♦ 

- 

4 

4 

4 

- 

41 

4 

- 

4 

- 

4 

4 

42 

- 

4 

- 

4 

- 

• 

43 

- 

- 

4 

- 

- 

- 

44 

4 

4 

- 

4 

4 

4 

45 

4 

4 

4 

- 

- 

- 

46 

- 

- 

- 

4 

4 

4 

47 

- 

4 

4 

- 

4 

4 

48 

♦ 

4 
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TABLE  A4 


CENTRAL  TRAINING  VEHICLE  CONFIGURATION 
FOR  THE  EIGHT  INDIVIDUALS  IN  GROUP  D  WHO  TRANSFERRED 
TO  THE  CENTRAL  TRANSFER  VEHICLE  CONFIGURATION 


GROUP  D 

Training  Vehicle  Configuration 

Training  Trials 

Participant 

CO  DL  TM  PT  CG 

TT 
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APPENDIX  B 

GENERAL  INSTRUCTIONS  READ  ALOUD  TO  ALL  PARTICIPANTS 
WHILE  THEY  SILENTLY  FOLLOWED  ON  A  COPY 


"These  are  the  instructions  for  the  task  you  will  be  asked  to 
perform.  I  will  read  the  instructions  aloud  while  you  follow  along 
silently. " 

"We  are  conducting  an  experiment  to  compare  various  types  of 
flight  training  displays.  We  know  that  you  are  unfamiliar  with  this 
type  of  task  but  you  will  find  that  you  do  better  after  a  while.  We 
are  only  interested  in  which  of  several  displays  i3  the  best  for 
training. " 


TASK 


"The  task  is  a  simplified  version  of  what  a  pilot  performs  while 
flying  an  airplane.  Centered  on  the  display  in  front  of  you  are  two 

symbols;  a  box  (  Q  )  which  is  the  target,  and  a  figure  ( -■  . )  that 

looks  something  like  a  rear  view  of  an  airplane.  By  moving  the  stick 
left  and  right,  you  will  be  able  to  control  the  bank  angle  of  the 
airplane  symbol  which  in  turn  controls  its  position  relative  to  the 
target.  Because  we  are  concerned  with  teaching  pilots  to  steer  a 
plane,  only  left-right  movements  of  the  stick  will  affect  the  simulated 
airplane  you  are  flying.  Your  task  will  be  to  keep  the  airplane  close 
to  the  target  as  much  of  the  time  as  you  can.  We  will  measure  how 
closely  you  are  able  to  follow  the  target  and  tell  you  your  score  at 
the  end  of  each  one-minute  trial." 


VARIATION  OF  TASKS 

"Different  participants  will  be  asked  to  perform  on  different 
types  of  displays.  You  may  be  asked  to  perform  on  displays  in  which 
only  the  target  symbol  will  move  left  and  right.  The  airplane  symbol 
will  rotate  but  will  always  remain  in  the  center  of  the  screen.  In 
other  types  of  displays,  both  the  airplane  symbol  and  the  target  symbol 
will  move  to  show  their  relative  positions.  In  either  case,  you  will 
be  able  to  control  the  position  of  the  'actual'  airplane  relative  to 
the  target  by  always  banking  or  turning  toward  the  moving  target 
symbol.  The  'circle'  that  usually  forms  the  'body'  of  the  airplane  can 
move  out  ahead  of  the  cross-shaped  part  of  the  airplane,  in  which  case 
it  predicts  where  the  airplane  will  be  a  moment  later.  Moderate  use  of 
the  control  stick  will  prevent  over-control  of  the  airplane." 
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Following  joint  reading  of  instructions  while  participant  remains 
in  control  chair,  experimenter  flies  one  demonstration  trial,  saying 
only  the  following: 

(1)  "Now  I’m  going  to  demonstrate  your  task." 

(2)  "I  am  trying  to  keep  the  airplane  as  close  to  the  target 

as  I  can." 

(3)  "Notice  that  to  do  this  I  always  keep  turning  the 

airplane  symbol  toward  the  target  by  banking  in  that 
direction." 

If  questions  are  asked,  they  are  to  be  answered  as  directly  and 
simply  as  possible.  Then  the  instructions  are  jointly  read  a  second 
time  beginning  with  the  section  labeled  TASK. 


PROCEDURE 

"You  will  fly  a  series  of  trials  that  are  approximately  one  minute 
in  length.  At  the  end  of  each  trial,  the  display  will  momentarily  go 
blank  and  then  reset  to  the  starting  position  with  all  the  symbols  in 
the  center  of  the  screen.  You  will  be  told  your  score  after  each 
trial.  Your  score  is  based  on  your  average  distance  from  the  target. 

The  lower  the  score,  the  better  the  performance.  Before  each  trial,  I 
will  signal  you  with  the  words  READY,  BEGIN.  The  information  we  obtain 
here  will  be  important  in  designing  pilot  training  systems.  All  we  ask 
is  that  you  do  your  best  during  the  entire  session.  Please  notify  me 
if  the  display  does  not  operate  like  I  said  it  would." 

"Are  there  any  questions?" 

"Will  you  please  explain  what  your  task  is  and  how  you  are  to 
accomplish  it?" 

(The  participant  replies  accordingly.) 

"If  there  are  no  further  questions,  wait  for  my  READY,  BEGIN 
signal." 

The  participant  will  now  fly  the  three  matching  trials  before 
which  the  experimenter  will  say, 

"You  will  now  fly  a  series  of  trials  with  an  experimental  display." 

Next,  the  participant  will  fly  the  two  masking  trials  and  the  block  of 
experimental  trials  before  which  the  experimenter  will  say, 

"You  will  now  fly  a  series  of  trials  with  a  different  experimental 
display." 

"READY,  BEGIN" 


I 
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APPENDIX  C 

PREANALYSIS  OF  EXPERIMENTAL  DATA 

Characteristics  of  data  often  determine  how  they  should  best  be 
analyzed.  Experimental  designs  alone  do  not  guarantee  high  quality 
information.  When  economical  multifactor  designs  are  employed,  the 
opportunities  for  finding  effects  due  to  chance  or  unintentional  biases 
makes  preanalysis  central  to  the  experimental  approach.  The  following 
are  some  of  the  more  important  results  of  a  preanalysis  of  the  data 
from  this  experiment. 

DATA  CHARACTERISTICS 

Relevant  characteristics  of  distributions  of  experimental  data 
samples  include  questions  of  normality,  homogeneity  of  variances, 
randomness  of  sampling,  and  sources  of  experimental  bias. 

Transformation 


Performance  was  originally  measured  in  root  mean  square  error  per 
trial.  The  question  is:  Should  we  analyze  the  data  a3  is  or  should 
they  first  be  transformed?  There  are  numerous  criteria  to  apply  before 
deciding  to  work  with  transformed  data;  one  rule  of  thumb  is  that  when 
the  deviations  of  minimum  and  maximum  scores  from  the  mean  differ  by 
more  than  two  to  one,  some  form  of  transformation  is  in  order. 

The  ratios  both  before  and  after  a  logarithmic  transformation  are 
shown  in  Table  Cl,  and  it  appears  that  (a)  a  transformation  of  some  kind 
was  desirable  and  (b)  the  log  transformation  served  its  purpose.  As  a 
methodological  note,  the  choice  of  the  logarithmic  transformation  was 
made  rationally  because  it  has  been  found  to  normalize  RMS  error  scores 
and  equalize  their  variances  in  the  past.  The  preferred  approach,  how¬ 
ever,  would  be  to  let  the  data  determine  the  choice  of  transformation, 
using  a  method  proposed  by  Draper  and  Hunter  (1969;  Simon,  1 98 1 )  for 
handling  multivariate  transformations. 

Central  Tendency  Measure 


Inspection  of  the  raw  data  revealed  that  individual  performances 
varied  considerably  from  trial  to  trial.  It  was  decided  to  combine  the 
data  over  several  trials  in  a  single  representative  score.  The  best 
index  of  central  tendency  of  a  number  of  trials  was  desired.  The 
question  is:  Should  this  measure  be  the  mean  or  median  of  each 
individual's  set  of  trials?  The  means  and  standard  deviations  of  the 
mean  and  median  scores  of  all  56  participants  in  the  experimental 
groups  for  these  sets  are  given  in  Table  C2.  Correlations  between  mean 
and  median  scores  for  the  56  participants  in  the  experimental  groups 
were:  training  performances,  first  five  trials,  0.990,  last  five 
trials,  0.996;  and  transfer  performances,  first  five  trials,  0.99^. 


TABLE  Cl 


COMPARISON  OF  DISTRIBUTIONS  OF  RAH  AND  TRANSFORMED  PERFORMANCE  SCORES 
FOR  THE  56  PARTICIPANTS  IN  GROUPS  A,  B,  C,  AND  D  COMBINED 


RMS  ERROR 

TRAINING  PHASE 

Transfer  phase 

Smallest 

15.96 

12.05 

Difference 

-35.44 

-36.70 

Mean 

51.40 

48.75 

Difference 

♦136.96 

♦423.31 

Largest 

188.36 

472.06 

Ratio  of  Absolute  Differences  3*86 

11.53 

1000  LOG  RMS 

Smallest 

1203 

1081 

Difference 

-508 

-607 

Mean 

1711 

1688 

Difference 

♦564 

♦986 

Largest 

2275 

2674 

Ratio  of  Absolute 

Differences  1.11 

1.62 
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TABLE  C2 

MEANS  AND  STANDARD  DEVIATIONS  OF  THE  MEANS  AND  MEDIANS 
OF  SCORES  FOR  BLOCKS  OF  FIVE  TRIALS  EACH  FOR  THE  56  PARTICIPANTS 
IN  GROUPS  A,  B,  C,  AND  D  COMBINED 


First  5  Second  5  First  5 

Training  Trials  Training  Trials  Transfer  Trials 


Mean 

Median 

Mean 

Median 

Mean 

Median 

Mean 

1782 

1775 

1718 

1711 

1697 

1688 

Standard 

Deviation 

246 

242 

237 

232 

372 

373 

It  could  be  argued  that  either  the  mean  or  median  of  a  set  of 
trials  would  be  an  acceptable  measure.  However,  because  sizable  dis¬ 
crepant  individual  scores  occasionally  occur  within  a  set  of  trials,  it 
was  decided  that  using  median  values  for  each  individual  would  provide 
the  best  protection  against  the  presence  of  occasional  outliers.  If 
there  were  no  greatly  discrepant  values,  means  and  medians  would  be 
essentially  the  same.  If  there  were  occasional  ones  for  some  partici¬ 
pants,  the  medians  would  provide  the  better  indication  of  central  ten¬ 
dency.  Thus  all  individual  scores  are  the  medians  of  sets  of  trials. 

Group  Matching 

The  experimental  design  consisted  of  seven  blocks  of  experimental 
conditions.  Three  blocks  are  those  for  the  three  experimental  groups 
associated  with  the  three  transfer  configurations.  One  block  was  at 
the  center  of  the  experimental  space.  The  remaining  three  represent 
the  three  control  groups.  The  question  is:  Did  these  seven  groups 
differ  initially  in  ability  on  this  task?  The  means  and  standard  devi¬ 
ations  of  each  group’s  initial  matching  scores  are  shown  in  Table  C3* 

A  test  of  the  differences  among  the  seven  means,  taking  group  size 
into  consideration,  yielded  an  F-ratio  with  6  and  73  degrees  of  freedom 
of  1.43,  a  value  that  might  be  expected  to  occur  by  chance  21  times  in 
100.  Bartlett's  test  of  the  homogeneity  of  variance  among  the  seven 
groups  yielded  an  approximate  F  of  1.43  and  a  £  of  0.20.  On  such  evi¬ 
dence  one  cannot  reject  the  hypothesis  that  the  groups  were  randomly 
drawn  from  a  homogeneous  population. 


75 


TABLE  C3 


MEANS  AND  STANDARD  DEVIATIONS  OF  MATCHING  SCORES  FOR  EACH  GROUP 


Experimental  Groups 

Center  Group 

Control  Groups 

(N  = 

16  each) 

(N  =  8) 

(N 

=  8  each) 

A 

B  C 

D 

E 

F  G 

Mean 

1704 

1697  1662 

1664 

1750 

1663  1688 

Standard 

Deviation 

98 

80  86 

36 

68 

60  92 

Bias  Check 


The  data  had  been  collected  by  three  experimenters  over  a  period 
of  several  weeks  and  at  different  times  of  the  day  or  evening.  Did  the 
different  experimenters,  time  of  day,  or  phase  of  the  data  collection 
effort  Introduce  any  detectable  biases  in  the  data?  Correlations  among 
those  three  factors  and  several  of  the  key  performance  measures  are 
shown  for  the  combined  data  in  Table  C4.  There  is  no  evidence  that  any 
bias  was  introduced  by  these  variables. 

CORRESPONDENCE  WITH  EXPECTATIONS 

One  means  of  examining  the  quality  of  empirical  data  is  to  see 
whether  they  correspond  with  prior  expectations.  Applications  of 
economical  multifactor  data  collection  plans  require  extensive  pre- 
experimental  analysis  to  anticipate  subsequent  results  based  on  what¬ 
ever  theory  and  fact  can  be  brought  to  bear  on  the  problem.  Explaining 
results  after  the  data  have  been  collected  provides  little  confidence 
in  their  interpretation  and  leads  to  the  traditional  sequence  of  exper¬ 
iments  to  verify  the  results  of  previous  experiments.  Gross  disagree¬ 
ments  between  expectations  and  data  warn  the  investigator  that  there  is 
something  wrong  either  with  the  theory  or  the  data. 

In  either  case,  discrepancies  serve  as  focal  points  for  a  more 
extensive  examination  of  data,  whereas  correspondence  between  expecta¬ 
tions  and  results  is  definitely  encouraging.  Nevertheless,  correspon¬ 
dence  remains  a  source  of  contentment  only  to  the  extent  that  the  data 
conform  to  several  criteria.  With  multifactor  designs,  however,  when 
complex  patterns  of  data  are  observed,  the  confidence  produced  from 
conforming  data  is  much  higher  than  would  be  the  case  with  simple 
designs  in  which  the  data  can  only  increase,  decrease,  or  reverse 
expectations  along  a  single  continuum  of  conditions.  In  this  study,  a 
limited  preanalysis  enabled  only  a  few  expectations  to  be  evaluated. 
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TABLE  C4 

CORRELATIONS  RELATING  POTENTIAL  SOURCES  OF  EXPERIMENTAL  BIAS 
TO  DIFFERENT  PERFORMANCE  SCORES  (N  =  56) 


Potential  Sources  of  Bias 


Scores 

Experimenters 

Experiment  Phase 

Time  of  Day 

Matching  Score 

♦  .017 

-.133 

-.003 

Training  Performance 

-.049 

♦  .028 

-.003 

Transfer  Performance 

-.127 

-.^29 

1 

o 

-4 

Transfer  Effect 

-.151 

♦  .028 

-.094 

Relative  Difficulty 

In  selecting  the  factor  levels,  pretests  were  made  to  confirm 
expectations  as  to  which  levels  would  lead  to  better  and  poorer  per¬ 
formances.  A  coded  -1  value  was  assigned  to  the  level  yielding  the 
poorer  performance.  Similarly,  estimates  were  made  regarding  the 
relative  difficulty  of  the  three  transfer  configurations,  coded  -1,  0, 
and  *1.  Other  less  design-oriented  predictions  were  made  regarding 
possible  interactions  among  factors,  but  these  preanalyses  were  scanty 
and  unsystematic.  In  retrospect,  our  inability  to  predict  factor 
interactions  during  training  and  transfer  was  due  considerably  to  the 
fact  that  even  experienced  investigators  have  had  little  or  no  oppor¬ 
tunity  to  see  the  effects  of  complex  relationships  and  interactions 
demanded  for  preanalyses  of  this  type. 

Let  us  see,  however,  to  what  extent  the  data  do  or  do  not  behave 
as  expected.  For  example:  Are  the  three  transfer  configurations  rela¬ 
tively  hard,  intermediate,  and  easy  as  predicted?  Median  performances 
for  the  three  control  groups  on  the  Hard,  Central,  and  Easy  system 
configurations,  respectively,  were:  2179,  1597,  and  1368,  with  lower 
scores  being  better.  The  order  was  as  expected,  although  the  Central 
configuration  was  not  midway  between  Hard  and  Easy. 


Amount  of  Learning 

Since  this  was  a  transfer  of  training  study,  it  is  appropriate  to 
ask:  To  what  extent  did  the  participants  learn?  Figure  C-1  shows  the 
learning  curves  created  by  connecting  the  means  of  scores  from  the 
matching,  masking,  first-five  training,  last-five  training,*  first-five 
transfer  and  last-five  transfer  trials  for  all  56  participants,  and 
additional  curves  for  each  of  the  four  groups  individually.  These 
curves  behave  remarkably  well,  and  in  accordance  with  expectations. 
However,  they  are  averages  and  therefore  sanitized.  During  training 
the  individual  curves  varied  considerably,  both  within  and  among  them¬ 
selves. 

The  mean  proportion  of  variance  accounted  for  by  the  residuals  for 
all  56  participants  after  that  due  to  linear,  quadratic,  and  cubic 
trends  was  isolated  is  0.62;  the  range  ran  from  .95  to  .13.  The  regu¬ 
larity  of  these  statistics  among  the  three  groups,  A,  B,  and  C  suggest 
that  the  equipment  configurations  rather  than  individuals  were  contrib¬ 
uting  to  this  within  learning  curve  variability.  Furthermore,  not  only 
were  the  learning  curves  somewhat  erratic,  but  also  there  was  consider¬ 
able  variability  between  curves  when  different  configurations  were  used. 
Some  exhibited  a  reasonable  amount  of  learning  and  some  showed  none. 
Still  the  correlations  among  the  main  factor  effects  and  the  linear 
performance  effects  during  training  were  all  low  (i.e.,  median,  .13; 
range  .11  to  .28;  N  =  56)  showing  that  the  factors  individually  did  not 
influence  how  much  learning  took  place  to  any  significant  degree. •• 

Additional  analyses  were  performed  on  the  data  from  the  primary 
experimental  groups.  For  each  participant  the  median  performance  of 
the  last  five  training  trials  (6-10  or  25-30,  depending  on  the  amount 
of  training)  waa  compared  with  that  from  the  first  five  training  trials. 
For  each  group,  the  difference  between  each  individual’s  scores  from 
those  two  sets  of  data  was  divided  by  the  median  performance  value  of 
all  first-five  training  trials  to  obtain  an  indication  of  each  indi¬ 
vidual's  percent  improvement.  Only  three  of  the  24  participants  who 
received  10  training  trials  showed  an  improvement  of  10%  or  better 
whereas  14  of  the  24  who  received  30  training  trials  improved  10%  or 
better.  Four  improved  more  than  20%  and  three  more  than  30%. 


*Last  five  training  trials  may  have  been  the  6th  to  10th,  16th  to  20th, 
or  26th  to  30th,  depending  on  the  group. 

**Note:  These  correlations  reflect  the  degree  of  linear  relationship 
that  existed.  If  a  curvilinear  relationship  existed,  it  might  not  be 
detected  and  would  in  fact  result  in  a  low  correlation  measure  of  the 
type  used  here.  No  plots  of  the  data  were  made. 
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N  of  each  group  in  porentheass 


Figure  Cl.  Mean  training  and  transfer  curves  for  Groups  A,  B,  C,  and  D 
singly  and  combined. 
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CONTROL 


These  results  suggest  that  the  performance  surface  Is  shallow. 
While  a  great  deal  of  pretransfer  learning  Is  not  an  absolute  pre¬ 
requisite  for  a  transfer  study,  lack  of  it  suggests  that  the  task  was 
either  too  easy  or  too  difficult.  This  does  not  negate  the  study's 
usefulness  as  a  demonstration  of  the  effectiveness  of  the  multifactor 
approach  to  transfer  of  training  research.  It  may  limit  the  interpre¬ 
tation  of  the  factor  effects  in  a  particular  study,  because  in  the  real 
world,  task  difficulty  levels  in  a  simulator  need  to  match  those  in  the 
^counterpart  airplane  if  results  of  comparative  studies  in  the  simulator 
are  to  be  trusted. 
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APPENDIX  D 
PERCENT  OF  TRANSFER 

The  transfer  effect  score  used  in  this  study  is  not  commonly 
employed  by  psychologists  to  measure  transfer  of  training  (see  Roscoe, 
1980,  Chapter  16).  Here,  only  differences  between  performance  scores 
on  the  transfer  vehicles  by  trained  and  control  participants  were  used 
in  the  main  analyses,  whereas  percent  transfer  scores  are  normally  used 
to  compare  devices.  This  latter  measure  is  obtained  by  dividing  the 
transfer  effect  score  by  the  control  group's  score  and  multiplying  by 
100,  a  simple  monotonic  transformation. 

Because  of  the  nature  of  the  task  in  this  study,  either  transfer 
effect  scores  or  percent  transfer  scores  might  have  been  used  to  demon¬ 
strate  the  practical  advantages  of  the  holistic  approach  to  transfer  of 
training  research.  However,  because  there  were  three  transfer  vehicle 
configurations  rather  than  one,  the  set  of  56  percent  transfer  scores 
was  not  the  result  of  a  simple  transformation  of  the  transfer  effect 
scores;  the  relationship  was  more  complex. 

For  example,  if  a  transfer  effect  score  in  Group  A  were  100  and 
another  in  Group  C  were  also  100,  the  two  percent  transfer  scores  would 
not  be  equal  since  the  divisor  for  Group  A  is  larger  than  the  divisor 
for  Group  C.  This  difference  would  not  change  the  Interpretation  of 
any  equipment  or  training  factor  (since  each  is  orthogonal  to  the 
transfer  vehicle  factor),  nor  their  interaction?.  However,  the  rela¬ 
tionships  between  the  experimental  factor  and  percent  transfer  scores 
across  the  three  different  transfer  vehicles  could  be  different  from 
those  between  the  experimental  factors  and  transfer  effect  scores. 

The  transfer  effect  scores  and  percent  transfer  scores  correlate 
.99,  which  for  all  practical  purposes  implies  identity.  This  suggests 
that  it  would  not  matter  which  measure  is  used,  at  least  for  the  re¬ 
sults  obtained  in  this  experiment.  The  percent  transfer  scores  and 
the  training  performance  scores  for  the  last  five  trials  correlate  .91. 
This  compares  with  the  correlation  of  .37  between  the  transfer  effect 
scores  and  the  same  training  performance.  The  difference  is  trivial. 

Table  D1  for  the  analysis  of  percent  transfer  scores  parallels 
Table  13  for  both  transfer  effect  and  transfer  performance  scores.  Two 
facts  stand  out  when  the  tables  are  compared.  First,  the  direction  of 
transfer  (whether  positive  or  negative)  is  the  same  for  every  source  of 
variance.  Second,  seven  effects  in  each  table  with  chance  probabili¬ 
ties  of  less  than  .05  were  the  same.  The  ten  largest  standard  regres¬ 
sion  coefficients  were  associated  with  the  same  sources  of  variance  in 
both  tables.  Although  some  differences  in  order  occurred,  the  results 
are  quite  similar  overall. 


TABLE  D1:  ANALYSIS  OF  TRANSFER  PERFORMANCE  EFFECTS: 
GROUPS  A,  B,  C,  AND  D 


Source 
of  Effect 

Regression 

Coefficient 

Standard 

Error 

Standardized 

Coefficient 

£ 

INTERCEPT 

-0.017 

Control  Order 

-0.011 

0.009 

-.12 

.229 

Display  Lag 

-0.020 

0.009 

-.22 

.03  2 

Tracking  Mode 

0.014 

0.009 

.16 

.117 

Prediction  Time 

0.006 

0.009 

.06 

.525 

Control  Gain 

0.017 

0.009 

.18 

.067 

Training  Trials 

0.001 

0.009 

.01 

.904 

TVC,  Linear  (XL) 

-0.039 

0.011 

-.35 

.001 

TVC,  Quadratic  (XQ) 

0.021 

0.007 

.28 

.007 

CO  x  DL 

0.002 

0.011 

.02 

.856 

CO  x  TM 

-0.014 

0.009 

-.14 

.165 

CO  x  PT 

0.014 

0.009 

.15 

.160 

CO  x  CG 

0.010 

0.009 

.11 

.292 

CO  X  TT 

0.005 

0.009 

.05 

.596 

CO  x  XL 

-0.017 

0.011 

-.15 

.131 

CO  X  XQ 

0.005 

0.006 

.08 

.410 

DL  x  TM 

-0.012 

0.009 

-.13 

.202 

DL  x  PT 

0.156 

0.009 

.16 

.131 

DL  x  CG 

0.003 

0.009 

.03 

.748 

DL  x  TT 

0.022 

0.009 

.23 

.031 

DL  x  XL 

-0.009 

0.011 

-.08 

.407 

DL  x  XQ 

0.004 

0.006 

.06 

.536 

TM  x  PT 

-0.017 

0.009 

-.19 

.080 

TM  x  CG 

-0.001 

0.011 

-.01 

.905 

TM  x  TT 

0.008 

0.009 

.09 

.387 

TM  x  XL 

-0.028 

0.011 

-.25 

.016 

TM  x  XQ 

0.013 

0.006 

.20 

.048 

PT  x  CG 

0.014 

0.009 

.15 

.147 

PT  x  TT 

-0.011 

0.011 

-.12 

.323 

PT  x  XL 

0.013 

0.011 

.11 

.261 

PT  x  XQ 

-0.008 

0.006 

-.12 

.207 

CG  x  TT 

-0.024 

0.009 

-.26 

.017 

CG  x  XL 

-0.005 

0.011 

-.05 

.617 

CG  x  XQ 

0.014 

0.006 

.21 

.035 

TT  x  XL 

0.006 

0.011 

.05 

.585 

TT  x  XQ 

0.000 

0.006 

.00 

.998 

A  stepwise  regression  analysis  of  the  percent  transfer  data 
brought  the  same  four  factors  into  the  equation  as  found  in  the  trans¬ 
fer  effects  analysis.  These  were,  in  order  of  importance,  XL  (.12), 

XQ  (.08),  DL  x  TT  (.08),  and  TM  x  XL  (.06),  with  the  proportion  of 
variance  each  accounted  for  shown  in  parentheses.  The  adjusted  multi¬ 
ple  I»  for  this  percent  transfer  equation  was  .29,  compared  to  .30  for 
the  equation  based  on  transfer  effects. 

In  summary,  although  the  findings  from  the  analyses  of  transfer 
effects  and  percent  transfer  scores  were  for  all  practical  purposes 
identical,  this  need  not  be  valid  in  general.  For  this  reason  the 
scores  analyzed  in  each  case  should  be  the  ones  of  greatest  interest  to 
the  user. 
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